You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/07/18 21:34:31 UTC
[GitHub] [iceberg] huaxingao opened a new pull request, #5302: Add SparkV2Filters
huaxingao opened a new pull request, #5302:
URL: https://github.com/apache/iceberg/pull/5302
This PR adds `SparkV2Filters` and `TestSparkV2Filters`. In next PR, I will make `SparkScanBuilder` implement `SupportsPushDownV2Filters` and add a test `TestV2FilteredScan`.
Currently, Spark still uses V2 Filter in `SupportsRuntimeFiltering` and `SupportsDelete`. I have a PR to add `SupportsRuntimeV2Filtering` and will have a PR to add V2 Filter in `SupportsDelete` in Spark. Before these support are released on Spark side, we can use `Filter.toV2` to convert V1 filter to V2 filter for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r954373492
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(attributeName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(attributeName);
+ } else {
+ return handleEqual(attributeName, value);
+ }
+ }
+
+ case IN:
+ return in(
Review Comment:
Changed. Thanks!
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(attributeName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(attributeName);
+ } else {
+ return handleEqual(attributeName, value);
+ }
+ }
+
+ case IN:
+ return in(
+ toColumnName(leftChild(predicate)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
Review Comment:
Fixed. Thanks!
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(attributeName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(attributeName);
+ } else {
+ return handleEqual(attributeName, value);
+ }
+ }
+
+ case IN:
+ return in(
+ toColumnName(leftChild(predicate)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
Review Comment:
Fixed. Thanks!
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(attributeName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(attributeName);
+ } else {
+ return handleEqual(attributeName, value);
+ }
+ }
+
+ case IN:
+ return in(
+ toColumnName(leftChild(predicate)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
Review Comment:
Fixed. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r923900082
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {
+ }
+
+ private static final Map<String, Expression.Operation> FILTERS = ImmutableMap
+ .<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (checkIfPredicateValid(predicate) == null) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(getAttrName(predicate.children()[0])));
+
+ case NOT_NULL:
+ return notNull(unquote(getAttrName(predicate.children()[0])));
+
+ case LT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThan(unquote(getAttrName(predicate.children()[0])),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(unquote(getAttrName(predicate.children()[1])),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThanOrEqual(unquote(getAttrName(predicate.children()[0])),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(unquote(getAttrName(predicate.children()[1])),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThan(unquote(getAttrName(predicate.children()[0])),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThan(unquote(getAttrName(predicate.children()[1])),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThanOrEqual(unquote(getAttrName(predicate.children()[0])),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(unquote(getAttrName(predicate.children()[1])),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value = null;
+ String attributeName = "";
+ if (predicate.children()[1] instanceof LiteralValue) {
+ attributeName = getAttrName(predicate.children()[0]);
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = getAttrName(predicate.children()[1]);
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value());
+ }
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(value,
+ "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(unquote(attributeName), value);
+ } else if (predicate.name().equals("<=>")) {
+ if (value == null) {
+ return isNull(unquote(attributeName));
+ } else {
+ return handleEqual(unquote(attributeName), value);
+ }
+ }
+ break;
+
+ case IN:
+ Object[] inValues = new Object[predicate.children().length - 1];
+ for (int i = 1; i < predicate.children().length; i++) {
+ inValues[i - 1] = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[i]).value());
+ }
+ return in(unquote(getAttrName(predicate.children()[0])),
+ Stream.of(inValues)
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in Iceberg
+ Object[] notInValues = new Object[childFilter.children().length - 1];
+ for (int i = 1; i < childFilter.children().length; i++) {
+ notInValues[i - 1] = convertUTF8StringIfNecessary(((LiteralValue) childFilter.children()[i]).value());
+ }
+ Expression notIn = notIn(unquote(getAttrName(childFilter.children()[0])),
+ Stream.of(notInValues).collect(Collectors.toList()));
+ return and(notNull(unquote(getAttrName(childFilter.children()[0]))), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND: {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR: {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH: {
+ return startsWith(unquote(getAttrName(predicate.children()[0])),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()).toString());
+ }
+ }
+ }
+
+ return null;
+ }
+
+ private static Object convertUTF8StringIfNecessary(Object value) {
+ if (value instanceof UTF8String) {
+ return ((UTF8String) value).toString();
+ }
+ return value;
+ }
+
+ private static Expression handleEqual(String attribute, Object value) {
+ if (NaNUtil.isNaN(value)) {
+ return isNaN(attribute);
+ } else {
+ return equal(attribute, value);
+ }
+ }
+
+ private static String unquote(String attributeName) {
+ Matcher matcher = BACKTICKS_PATTERN.matcher(attributeName);
+ return matcher.replaceAll("$2");
+ }
+
+ private static boolean hasNoInFilter(Predicate predicate) {
+ Expression.Operation op = FILTERS.get(predicate.name());
+
+ if (op != null) {
+ switch (op) {
+ case AND:
+ And andPredicate = (And) predicate;
+ return hasNoInFilter(andPredicate.left()) && hasNoInFilter(andPredicate.right());
+ case OR:
+ Or orPredicate = (Or) predicate;
+ return hasNoInFilter(orPredicate.left()) && hasNoInFilter(orPredicate.right());
+ case NOT:
+ Not notPredicate = (Not) predicate;
+ return hasNoInFilter(notPredicate.child());
+ case IN:
+ return false;
+ default:
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ private static String getAttrName(org.apache.spark.sql.connector.expressions.Expression expr) {
+ return String.join(".", ((NamedReference) expr).fieldNames());
+ }
+
+ @SuppressWarnings("checkstyle:CyclomaticComplexity")
+ private static Predicate checkIfPredicateValid(Predicate predicate) {
Review Comment:
I think Spark should have this check to make sure all the V2 Filters are in valid format. Otherwise, this kind of invalid [filter](https://github.com/apache/iceberg/blob/master/spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java#L433) will get through.
I used to have this check on Spark side, but somehow it got removed during refactor. I will add this check back. Before this is available, I will have this check here. I will remove once this is in Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r946181165
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (checkIfPredicateValid(predicate) == null) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[0].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[0].toString()));
+
+ case LT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value = null;
Review Comment:
Nit: looks like we dont need null and "" (java defaults)
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (checkIfPredicateValid(predicate) == null) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[0].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[0].toString()));
+
+ case LT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value = null;
+ String attributeName = "";
+ if (predicate.children()[1] instanceof LiteralValue) {
+ attributeName = predicate.children()[0].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = predicate.children()[1].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value());
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(unquote(attributeName), value);
+ } else if (predicate.name().equals("<=>")) {
+ if (value == null) {
+ return isNull(unquote(attributeName));
+ } else {
+ return handleEqual(unquote(attributeName), value);
+ }
+ }
+ break;
+
+ case IN:
+ Object[] inValues = new Object[predicate.children().length - 1];
+ for (int i = 1; i < predicate.children().length; i++) {
+ inValues[i - 1] =
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[i]).value());
+ }
+ return in(
+ unquote(predicate.children()[0].toString()),
+ Stream.of(inValues).filter(Objects::nonNull).collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Object[] notInValues = new Object[childFilter.children().length - 1];
+ for (int i = 1; i < childFilter.children().length; i++) {
+ notInValues[i - 1] =
+ convertUTF8StringIfNecessary(((LiteralValue) childFilter.children()[i]).value());
+ }
+ Expression notIn =
+ notIn(
+ unquote(childFilter.children()[0].toString()),
+ Stream.of(notInValues).collect(Collectors.toList()));
+ return and(notNull(unquote(childFilter.children()[0].toString())), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ {
+ return startsWith(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value())
+ .toString());
+ }
+ }
+ }
+
+ return null;
+ }
+
+ private static Object convertUTF8StringIfNecessary(Object value) {
+ if (value instanceof UTF8String) {
+ return ((UTF8String) value).toString();
+ }
+ return value;
+ }
+
+ private static Expression handleEqual(String attribute, Object value) {
+ if (NaNUtil.isNaN(value)) {
+ return isNaN(attribute);
+ } else {
+ return equal(attribute, value);
+ }
+ }
+
+ private static String unquote(String attributeName) {
+ Matcher matcher = BACKTICKS_PATTERN.matcher(attributeName);
+ return matcher.replaceAll("$2");
+ }
+
+ private static boolean hasNoInFilter(Predicate predicate) {
+ Expression.Operation op = FILTERS.get(predicate.name());
+
+ if (op != null) {
+ switch (op) {
+ case AND:
+ And andPredicate = (And) predicate;
+ return hasNoInFilter(andPredicate.left()) && hasNoInFilter(andPredicate.right());
+ case OR:
+ Or orPredicate = (Or) predicate;
+ return hasNoInFilter(orPredicate.left()) && hasNoInFilter(orPredicate.right());
+ case NOT:
+ Not notPredicate = (Not) predicate;
+ return hasNoInFilter(notPredicate.child());
+ case IN:
+ return false;
+ default:
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ @SuppressWarnings("checkstyle:CyclomaticComplexity")
+ private static Predicate checkIfPredicateValid(Predicate predicate) {
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case IS_NULL:
+ case NOT_NULL:
+ if (!(predicate.children()[0] instanceof NamedReference)) {
+ return null;
+ }
+ return predicate;
+
+ case LT:
+ case LT_EQ:
+ case GT:
+ case GT_EQ:
+ case EQ:
+ if (predicate.children().length != 2
Review Comment:
Just a personal preference but maybe its easier to read (if we go with the boolean return value approach):
```
if (predicate.children.length() != 2) {
return false;
}
return (predicate.children[0] instanceof NamedReference && predicate.children[1] instanceof LiteralReference) ||
(predicate.children[0] instanceof LiteralRefernce && predicate.children[1] instanceof NamedReference]);
```
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (checkIfPredicateValid(predicate) == null) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[0].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[0].toString()));
+
+ case LT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value = null;
+ String attributeName = "";
+ if (predicate.children()[1] instanceof LiteralValue) {
+ attributeName = predicate.children()[0].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = predicate.children()[1].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value());
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(unquote(attributeName), value);
+ } else if (predicate.name().equals("<=>")) {
+ if (value == null) {
+ return isNull(unquote(attributeName));
+ } else {
+ return handleEqual(unquote(attributeName), value);
+ }
+ }
+ break;
+
+ case IN:
+ Object[] inValues = new Object[predicate.children().length - 1];
+ for (int i = 1; i < predicate.children().length; i++) {
+ inValues[i - 1] =
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[i]).value());
+ }
+ return in(
+ unquote(predicate.children()[0].toString()),
+ Stream.of(inValues).filter(Objects::nonNull).collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Object[] notInValues = new Object[childFilter.children().length - 1];
+ for (int i = 1; i < childFilter.children().length; i++) {
+ notInValues[i - 1] =
+ convertUTF8StringIfNecessary(((LiteralValue) childFilter.children()[i]).value());
+ }
+ Expression notIn =
+ notIn(
+ unquote(childFilter.children()[0].toString()),
+ Stream.of(notInValues).collect(Collectors.toList()));
+ return and(notNull(unquote(childFilter.children()[0].toString())), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ {
+ return startsWith(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value())
+ .toString());
+ }
+ }
+ }
+
+ return null;
+ }
+
+ private static Object convertUTF8StringIfNecessary(Object value) {
+ if (value instanceof UTF8String) {
+ return ((UTF8String) value).toString();
+ }
+ return value;
+ }
+
+ private static Expression handleEqual(String attribute, Object value) {
+ if (NaNUtil.isNaN(value)) {
+ return isNaN(attribute);
+ } else {
+ return equal(attribute, value);
+ }
+ }
+
+ private static String unquote(String attributeName) {
+ Matcher matcher = BACKTICKS_PATTERN.matcher(attributeName);
+ return matcher.replaceAll("$2");
+ }
+
+ private static boolean hasNoInFilter(Predicate predicate) {
+ Expression.Operation op = FILTERS.get(predicate.name());
+
+ if (op != null) {
+ switch (op) {
+ case AND:
+ And andPredicate = (And) predicate;
+ return hasNoInFilter(andPredicate.left()) && hasNoInFilter(andPredicate.right());
+ case OR:
+ Or orPredicate = (Or) predicate;
+ return hasNoInFilter(orPredicate.left()) && hasNoInFilter(orPredicate.right());
+ case NOT:
+ Not notPredicate = (Not) predicate;
+ return hasNoInFilter(notPredicate.child());
+ case IN:
+ return false;
+ default:
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ @SuppressWarnings("checkstyle:CyclomaticComplexity")
+ private static Predicate checkIfPredicateValid(Predicate predicate) {
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case IS_NULL:
+ case NOT_NULL:
+ if (!(predicate.children()[0] instanceof NamedReference)) {
+ return null;
+ }
+ return predicate;
+
+ case LT:
+ case LT_EQ:
+ case GT:
+ case GT_EQ:
+ case EQ:
+ if (predicate.children().length != 2
+ || !(((predicate.children()[1] instanceof LiteralValue
+ && predicate.children()[0] instanceof NamedReference))
+ || ((predicate.children()[0] instanceof LiteralValue
+ && predicate.children()[1] instanceof NamedReference)))) {
+ return null;
+ }
+ return predicate;
+
+ case IN:
+ if (!(predicate.children()[0] instanceof NamedReference)) {
+ return null;
+ } else {
+ for (int i = 1; i < predicate.children().length; i++) {
+ if (!(predicate.children()[i] instanceof LiteralValue)) {
+ return null;
+ }
+ }
+ return predicate;
+ }
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ return checkIfPredicateValid(notFilter.child());
+
+ case AND:
+ And andFilter = (And) predicate;
+ if (checkIfPredicateValid(andFilter.left()) == null
Review Comment:
Same if we change, we could have an easier to read:
return valid(andFilter.left) && valid(andFilter.right);
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (checkIfPredicateValid(predicate) == null) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[0].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[0].toString()));
+
+ case LT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value = null;
+ String attributeName = "";
+ if (predicate.children()[1] instanceof LiteralValue) {
+ attributeName = predicate.children()[0].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = predicate.children()[1].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value());
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(unquote(attributeName), value);
+ } else if (predicate.name().equals("<=>")) {
+ if (value == null) {
+ return isNull(unquote(attributeName));
+ } else {
+ return handleEqual(unquote(attributeName), value);
+ }
+ }
+ break;
+
+ case IN:
+ Object[] inValues = new Object[predicate.children().length - 1];
+ for (int i = 1; i < predicate.children().length; i++) {
Review Comment:
Why not just make this a stream (as the next line anyway converts it to stream). Ie, predicate.children.stream().map(c -> convertUTF8StringIfNecessary(...))
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (checkIfPredicateValid(predicate) == null) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[0].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[0].toString()));
+
+ case LT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
Review Comment:
I'm not very familiar but In the old SparkFilters code there used to be a 'convertLiteral' functionality. Do we need that for literals?
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (checkIfPredicateValid(predicate) == null) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[0].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[0].toString()));
+
+ case LT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value = null;
+ String attributeName = "";
+ if (predicate.children()[1] instanceof LiteralValue) {
+ attributeName = predicate.children()[0].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = predicate.children()[1].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value());
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(unquote(attributeName), value);
+ } else if (predicate.name().equals("<=>")) {
+ if (value == null) {
+ return isNull(unquote(attributeName));
+ } else {
+ return handleEqual(unquote(attributeName), value);
+ }
+ }
+ break;
+
+ case IN:
+ Object[] inValues = new Object[predicate.children().length - 1];
+ for (int i = 1; i < predicate.children().length; i++) {
+ inValues[i - 1] =
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[i]).value());
+ }
+ return in(
+ unquote(predicate.children()[0].toString()),
+ Stream.of(inValues).filter(Objects::nonNull).collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Object[] notInValues = new Object[childFilter.children().length - 1];
+ for (int i = 1; i < childFilter.children().length; i++) {
+ notInValues[i - 1] =
+ convertUTF8StringIfNecessary(((LiteralValue) childFilter.children()[i]).value());
+ }
+ Expression notIn =
+ notIn(
+ unquote(childFilter.children()[0].toString()),
+ Stream.of(notInValues).collect(Collectors.toList()));
+ return and(notNull(unquote(childFilter.children()[0].toString())), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ {
+ return startsWith(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value())
+ .toString());
+ }
+ }
+ }
+
+ return null;
+ }
+
+ private static Object convertUTF8StringIfNecessary(Object value) {
+ if (value instanceof UTF8String) {
+ return ((UTF8String) value).toString();
+ }
+ return value;
+ }
+
+ private static Expression handleEqual(String attribute, Object value) {
+ if (NaNUtil.isNaN(value)) {
+ return isNaN(attribute);
+ } else {
+ return equal(attribute, value);
+ }
+ }
+
+ private static String unquote(String attributeName) {
+ Matcher matcher = BACKTICKS_PATTERN.matcher(attributeName);
+ return matcher.replaceAll("$2");
+ }
+
+ private static boolean hasNoInFilter(Predicate predicate) {
+ Expression.Operation op = FILTERS.get(predicate.name());
+
+ if (op != null) {
+ switch (op) {
+ case AND:
+ And andPredicate = (And) predicate;
+ return hasNoInFilter(andPredicate.left()) && hasNoInFilter(andPredicate.right());
+ case OR:
+ Or orPredicate = (Or) predicate;
+ return hasNoInFilter(orPredicate.left()) && hasNoInFilter(orPredicate.right());
+ case NOT:
+ Not notPredicate = (Not) predicate;
+ return hasNoInFilter(notPredicate.child());
+ case IN:
+ return false;
+ default:
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ @SuppressWarnings("checkstyle:CyclomaticComplexity")
+ private static Predicate checkIfPredicateValid(Predicate predicate) {
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case IS_NULL:
+ case NOT_NULL:
+ if (!(predicate.children()[0] instanceof NamedReference)) {
+ return null;
+ }
+ return predicate;
+
+ case LT:
+ case LT_EQ:
+ case GT:
+ case GT_EQ:
+ case EQ:
+ if (predicate.children().length != 2
+ || !(((predicate.children()[1] instanceof LiteralValue
+ && predicate.children()[0] instanceof NamedReference))
+ || ((predicate.children()[0] instanceof LiteralValue
+ && predicate.children()[1] instanceof NamedReference)))) {
+ return null;
+ }
+ return predicate;
+
+ case IN:
+ if (!(predicate.children()[0] instanceof NamedReference)) {
+ return null;
+ } else {
+ for (int i = 1; i < predicate.children().length; i++) {
Review Comment:
Also optional style suggestion, can use predicate.children.stream().allMatch()
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (checkIfPredicateValid(predicate) == null) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[0].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[0].toString()));
+
+ case LT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value = null;
+ String attributeName = "";
+ if (predicate.children()[1] instanceof LiteralValue) {
+ attributeName = predicate.children()[0].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = predicate.children()[1].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value());
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(unquote(attributeName), value);
+ } else if (predicate.name().equals("<=>")) {
+ if (value == null) {
+ return isNull(unquote(attributeName));
+ } else {
+ return handleEqual(unquote(attributeName), value);
+ }
+ }
+ break;
+
+ case IN:
+ Object[] inValues = new Object[predicate.children().length - 1];
+ for (int i = 1; i < predicate.children().length; i++) {
+ inValues[i - 1] =
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[i]).value());
+ }
+ return in(
+ unquote(predicate.children()[0].toString()),
+ Stream.of(inValues).filter(Objects::nonNull).collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Object[] notInValues = new Object[childFilter.children().length - 1];
+ for (int i = 1; i < childFilter.children().length; i++) {
Review Comment:
Same comment (use a stream)
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (checkIfPredicateValid(predicate) == null) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[0].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[0].toString()));
+
+ case LT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value = null;
+ String attributeName = "";
+ if (predicate.children()[1] instanceof LiteralValue) {
+ attributeName = predicate.children()[0].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = predicate.children()[1].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value());
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(unquote(attributeName), value);
+ } else if (predicate.name().equals("<=>")) {
+ if (value == null) {
+ return isNull(unquote(attributeName));
+ } else {
+ return handleEqual(unquote(attributeName), value);
+ }
+ }
+ break;
+
+ case IN:
+ Object[] inValues = new Object[predicate.children().length - 1];
+ for (int i = 1; i < predicate.children().length; i++) {
+ inValues[i - 1] =
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[i]).value());
+ }
+ return in(
+ unquote(predicate.children()[0].toString()),
+ Stream.of(inValues).filter(Objects::nonNull).collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Object[] notInValues = new Object[childFilter.children().length - 1];
+ for (int i = 1; i < childFilter.children().length; i++) {
+ notInValues[i - 1] =
+ convertUTF8StringIfNecessary(((LiteralValue) childFilter.children()[i]).value());
+ }
+ Expression notIn =
+ notIn(
+ unquote(childFilter.children()[0].toString()),
+ Stream.of(notInValues).collect(Collectors.toList()));
+ return and(notNull(unquote(childFilter.children()[0].toString())), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ {
+ return startsWith(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value())
+ .toString());
+ }
+ }
+ }
+
+ return null;
+ }
+
+ private static Object convertUTF8StringIfNecessary(Object value) {
+ if (value instanceof UTF8String) {
+ return ((UTF8String) value).toString();
+ }
+ return value;
+ }
+
+ private static Expression handleEqual(String attribute, Object value) {
+ if (NaNUtil.isNaN(value)) {
+ return isNaN(attribute);
+ } else {
+ return equal(attribute, value);
+ }
+ }
+
+ private static String unquote(String attributeName) {
+ Matcher matcher = BACKTICKS_PATTERN.matcher(attributeName);
+ return matcher.replaceAll("$2");
+ }
+
+ private static boolean hasNoInFilter(Predicate predicate) {
+ Expression.Operation op = FILTERS.get(predicate.name());
+
+ if (op != null) {
+ switch (op) {
+ case AND:
+ And andPredicate = (And) predicate;
+ return hasNoInFilter(andPredicate.left()) && hasNoInFilter(andPredicate.right());
+ case OR:
+ Or orPredicate = (Or) predicate;
+ return hasNoInFilter(orPredicate.left()) && hasNoInFilter(orPredicate.right());
+ case NOT:
+ Not notPredicate = (Not) predicate;
+ return hasNoInFilter(notPredicate.child());
+ case IN:
+ return false;
+ default:
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ @SuppressWarnings("checkstyle:CyclomaticComplexity")
+ private static Predicate checkIfPredicateValid(Predicate predicate) {
Review Comment:
I think we usually prefer shorter method names, so maybe valid(Predicate predicate) is better? Also, kind of related to the previous comment, why do we have the return value be a boolean instead of predicate? (valid seems to be a true/false question)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
kbendick commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r948670754
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put(TRUE, Expression.Operation.TRUE)
+ .put(FALSE, Expression.Operation.FALSE)
+ .put(EQ, Expression.Operation.EQ)
+ .put(EQ_NULL_SAFE, Expression.Operation.EQ)
+ .put(GT, Expression.Operation.GT)
+ .put(GT_EQ, Expression.Operation.GT_EQ)
+ .put(LT, Expression.Operation.LT)
+ .put(LT_EQ, Expression.Operation.LT_EQ)
+ .put(IN, Expression.Operation.IN)
+ .put(IS_NULL, Expression.Operation.IS_NULL)
+ .put(NOT_NULL, Expression.Operation.NOT_NULL)
+ .put(AND, Expression.Operation.AND)
+ .put(OR, Expression.Operation.OR)
+ .put(NOT, Expression.Operation.NOT)
+ .put(STARTS_WITH, Expression.Operation.STARTS_WITH)
+ .build();
+
+ private static final int FIRST_ORDINAL = 0;
+ private static final int SECOND_ORDINAL = 1;
Review Comment:
I know I suggested first_ordinal and second_ordinal, but if you’re not in love with those, after looking at it further I think “left child” and “right child” might make sense too.
But I also don’t see any need to bike shed / change further just for the sake of changing. This is plenty readable as-is for me. Just throwing it out there as another idea.
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
Review Comment:
Nit: Maybe move the constructor before or after all of the private static constants?
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put(TRUE, Expression.Operation.TRUE)
+ .put(FALSE, Expression.Operation.FALSE)
+ .put(EQ, Expression.Operation.EQ)
+ .put(EQ_NULL_SAFE, Expression.Operation.EQ)
+ .put(GT, Expression.Operation.GT)
+ .put(GT_EQ, Expression.Operation.GT_EQ)
+ .put(LT, Expression.Operation.LT)
+ .put(LT_EQ, Expression.Operation.LT_EQ)
+ .put(IN, Expression.Operation.IN)
+ .put(IS_NULL, Expression.Operation.IS_NULL)
+ .put(NOT_NULL, Expression.Operation.NOT_NULL)
+ .put(AND, Expression.Operation.AND)
+ .put(OR, Expression.Operation.OR)
+ .put(NOT, Expression.Operation.NOT)
+ .put(STARTS_WITH, Expression.Operation.STARTS_WITH)
+ .build();
+
+ private static final int FIRST_ORDINAL = 0;
+ private static final int SECOND_ORDINAL = 1;
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
+
+ case LT:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[SECOND_ORDINAL]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ attributeName = predicate.children()[FIRST_ORDINAL].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = predicate.children()[SECOND_ORDINAL].toString();
+ value =
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value());
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
Review Comment:
Question: will this fail the query?
So if somebody writes `select a, b from foo where c = null`, the query will throw? Would we be better off logging a warning and then using `Expressions.alwaysFalse()` (or null), or is throwing more user friendly since probably they didn’t mean for that to be the query — the query basically will not match anything right?
Genuinely asking as I’m not sure if people use this pattern at all. You’re right it seems like a mistake.
Maybe we should add a unit test that this throws the specified exception?
I’m genuinely not sure which is best so just asking. I’m good with it as is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r953219741
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
Review Comment:
Missing validation that we have 2 children? It is probably because we need to access the first child in IN predicates, which may have more than 2 children. If so, we could do something like this for IN predicates without removing the required validation in `leftChild`.
```
(NamedReference) predicate.children()[0])
```
Or you could define another helper function.
```
@SuppressWarnings("unchecked")
private static <T> T childAtIndex(Predicate predicate, int index) {
return (T) predicate.children()[index];
}
```
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
Review Comment:
These methods look good. Can we move these private methods below public ones, please?
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(attributeName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(attributeName);
+ } else {
+ return handleEqual(attributeName, value);
+ }
+ }
+
+ case IN:
+ return in(
+ toColumnName(leftChild(predicate)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Expression notIn =
+ notIn(
+ toColumnName(leftChild(childFilter)),
+ Arrays.stream(childFilter.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(toColumnName(leftChild(childFilter))), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ return startsWith(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)).toString());
+ }
+ }
+
+ return null;
+ }
+
+ private static Object convertUTF8StringIfNecessary(Literal literal) {
Review Comment:
nit: Just `convertLiteral`?
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(attributeName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(attributeName);
+ } else {
+ return handleEqual(attributeName, value);
+ }
+ }
+
+ case IN:
+ return in(
+ toColumnName(leftChild(predicate)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
Review Comment:
nit: `notPredicate`?
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put(TRUE, Expression.Operation.TRUE)
+ .put(FALSE, Expression.Operation.FALSE)
+ .put(EQ, Expression.Operation.EQ)
+ .put(EQ_NULL_SAFE, Expression.Operation.EQ)
+ .put(GT, Expression.Operation.GT)
+ .put(GT_EQ, Expression.Operation.GT_EQ)
+ .put(LT, Expression.Operation.LT)
+ .put(LT_EQ, Expression.Operation.LT_EQ)
+ .put(IN, Expression.Operation.IN)
+ .put(IS_NULL, Expression.Operation.IS_NULL)
+ .put(NOT_NULL, Expression.Operation.NOT_NULL)
+ .put(AND, Expression.Operation.AND)
+ .put(OR, Expression.Operation.OR)
+ .put(NOT, Expression.Operation.NOT)
+ .put(STARTS_WITH, Expression.Operation.STARTS_WITH)
+ .build();
+
+ private static final int FIRST_ORDINAL = 0;
+ private static final int SECOND_ORDINAL = 1;
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
Review Comment:
I see. It makes sense but what do you think about adapting the conversion logic a bit instead of doing two passes? We anyway need to check if left or right child is literal/reference. You could define helper methods like this.
```
private static boolean isRef(org.apache.spark.sql.connector.expressions.Expression expr) {
return expr instanceof NamedReference;
}
private static boolean isLiteral(org.apache.spark.sql.connector.expressions.Expression expr) {
return expr instanceof Literal;
}
```
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
Review Comment:
To have this as one pass, we could have:
```
if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
String columnName = toColumnName(leftChild(predicate));
return lessThan(columnName, convertLiteral(rightChild(predicate)));
} else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
String columnName = toColumnName(rightChild(predicate));
return greaterThan(columnName, convertLiteral(leftChild(predicate)));
} else {
return null;
}
```
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
Review Comment:
To have this as one pass, we could have:
```
return isRef(child(predicate)) ? isNull(toColumnName(child(predicate))) : null;
```
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(attributeName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(attributeName);
+ } else {
+ return handleEqual(attributeName, value);
+ }
+ }
+
+ case IN:
+ return in(
+ toColumnName(leftChild(predicate)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
Review Comment:
nit: What about replacing `equivalent` with `equal` so that the last line of the comment would fit on one line?
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(attributeName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(attributeName);
+ } else {
+ return handleEqual(attributeName, value);
+ }
+ }
+
+ case IN:
+ return in(
+ toColumnName(leftChild(predicate)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Expression notIn =
+ notIn(
+ toColumnName(leftChild(childFilter)),
+ Arrays.stream(childFilter.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
Review Comment:
nit: `Literal<?>` to avoid warnings about raw usage of parameterized types?
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
Review Comment:
nit: Shall we use `EQ` constant defined above?
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(attributeName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(attributeName);
+ } else {
+ return handleEqual(attributeName, value);
+ }
+ }
+
+ case IN:
+ return in(
+ toColumnName(leftChild(predicate)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
Review Comment:
nit: Use `IN` constant defined above?
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(attributeName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(attributeName);
+ } else {
+ return handleEqual(attributeName, value);
+ }
+ }
+
+ case IN:
+ return in(
+ toColumnName(leftChild(predicate)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Expression notIn =
+ notIn(
+ toColumnName(leftChild(childFilter)),
+ Arrays.stream(childFilter.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(toColumnName(leftChild(childFilter))), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ return startsWith(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)).toString());
Review Comment:
Hm, do we need `toString` or a cast would work too to be sure we have a string?
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(attributeName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(attributeName);
+ } else {
+ return handleEqual(attributeName, value);
+ }
+ }
+
+ case IN:
+ return in(
+ toColumnName(leftChild(predicate)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Expression notIn =
+ notIn(
+ toColumnName(leftChild(childFilter)),
+ Arrays.stream(childFilter.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(toColumnName(leftChild(childFilter))), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ return startsWith(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)).toString());
+ }
+ }
+
+ return null;
+ }
+
+ private static Object convertUTF8StringIfNecessary(Literal literal) {
Review Comment:
nit: `Literal` -> `Literal<?>`
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(attributeName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(attributeName);
+ } else {
+ return handleEqual(attributeName, value);
+ }
+ }
+
+ case IN:
+ return in(
Review Comment:
To have this as one pass, we could have:
```
if (isSupportedInPredicate(predicate)) {
return in(
toColumnName((NamedReference) predicate.children()[0]),
Arrays.stream(predicate.children())
.skip(1)
.map(child -> convertLiteral(((Literal<?>) child)))
.filter(Objects::nonNull)
.collect(Collectors.toList()));
} else {
return null;
}
```
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(attributeName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(attributeName);
+ } else {
+ return handleEqual(attributeName, value);
+ }
+ }
+
+ case IN:
+ return in(
+ toColumnName(leftChild(predicate)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
Review Comment:
nit: `childPredicate`?
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(attributeName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(attributeName);
+ } else {
+ return handleEqual(attributeName, value);
+ }
+ }
+
+ case IN:
+ return in(
+ toColumnName(leftChild(predicate)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Expression notIn =
+ notIn(
+ toColumnName(leftChild(childFilter)),
+ Arrays.stream(childFilter.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(toColumnName(leftChild(childFilter))), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ return startsWith(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)).toString());
+ }
+ }
+
+ return null;
+ }
+
+ private static Object convertUTF8StringIfNecessary(Literal literal) {
+ if (literal.value() instanceof UTF8String) {
+ return ((UTF8String) literal.value()).toString();
+ }
+ return literal.value();
+ }
+
+ private static Expression handleEqual(String attribute, Object value) {
+ if (NaNUtil.isNaN(value)) {
+ return isNaN(attribute);
+ } else {
+ return equal(attribute, value);
+ }
+ }
+
+ private static boolean hasNoInFilter(Predicate predicate) {
+ Operation op = FILTERS.get(predicate.name());
+
+ if (op != null) {
+ switch (op) {
+ case AND:
+ And andPredicate = (And) predicate;
+ return hasNoInFilter(andPredicate.left()) && hasNoInFilter(andPredicate.right());
+ case OR:
+ Or orPredicate = (Or) predicate;
+ return hasNoInFilter(orPredicate.left()) && hasNoInFilter(orPredicate.right());
+ case NOT:
+ Not notPredicate = (Not) predicate;
+ return hasNoInFilter(notPredicate.child());
+ case IN:
+ return false;
+ default:
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ @SuppressWarnings("checkstyle:CyclomaticComplexity")
+ private static boolean valid(Predicate predicate) {
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case IS_NULL:
+ case NOT_NULL:
+ return child(predicate) instanceof NamedReference;
+
+ case LT:
+ case LT_EQ:
+ case GT:
+ case GT_EQ:
+ case EQ:
+ if (predicate.children().length != 2) {
+ return false;
+ }
+ return namedRefCompLit(predicate) || litCompNamedRef(predicate);
+
+ case IN:
+ if (!(leftChild(predicate) instanceof NamedReference)) {
+ return false;
+ } else {
+ return Arrays.stream(predicate.children())
+ .skip(1)
+ .allMatch(val -> val instanceof Literal);
+ }
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ return valid(notFilter.child());
+
+ case AND:
+ And andFilter = (And) predicate;
+ return valid(andFilter.left()) && valid(andFilter.right());
+
+ case OR:
+ Or orFilter = (Or) predicate;
+ return valid(orFilter.left()) && valid(orFilter.right());
+
+ case STARTS_WITH:
+ if (predicate.children().length != 2) {
+ return false;
+ }
+ return namedRefCompLit(predicate)
+ && ((Literal<?>) rightChild(predicate)).value() instanceof String;
Review Comment:
Hm, was it actually correct? Wouldn't it be `UTF8String`?
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/TestSparkV2Filters.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.temporal.ChronoUnit;
+import java.util.Map;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DataTypes;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestSparkV2Filters {
+
+ @Test
+ public void testV2Filters() {
Review Comment:
Can we add a test for `startsWith`? I think the was a problem in the conversion and tests did not catch it.
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put(TRUE, Expression.Operation.TRUE)
+ .put(FALSE, Expression.Operation.FALSE)
+ .put(EQ, Expression.Operation.EQ)
+ .put(EQ_NULL_SAFE, Expression.Operation.EQ)
+ .put(GT, Expression.Operation.GT)
+ .put(GT_EQ, Expression.Operation.GT_EQ)
+ .put(LT, Expression.Operation.LT)
+ .put(LT_EQ, Expression.Operation.LT_EQ)
+ .put(IN, Expression.Operation.IN)
+ .put(IS_NULL, Expression.Operation.IS_NULL)
+ .put(NOT_NULL, Expression.Operation.NOT_NULL)
+ .put(AND, Expression.Operation.AND)
+ .put(OR, Expression.Operation.OR)
+ .put(NOT, Expression.Operation.NOT)
+ .put(STARTS_WITH, Expression.Operation.STARTS_WITH)
+ .build();
+
+ private static final int FIRST_ORDINAL = 0;
+ private static final int SECOND_ORDINAL = 1;
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
+
+ case LT:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[SECOND_ORDINAL]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ attributeName = predicate.children()[FIRST_ORDINAL].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = predicate.children()[SECOND_ORDINAL].toString();
+ value =
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value());
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(unquote(attributeName), value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(unquote(attributeName));
+ } else {
+ return handleEqual(unquote(attributeName), value);
+ }
+ }
+
+ case IN:
+ return in(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((LiteralValue) val).value()))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Expression notIn =
+ notIn(
+ unquote(childFilter.children()[FIRST_ORDINAL].toString()),
+ Arrays.stream(childFilter.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((LiteralValue) val).value()))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(unquote(childFilter.children()[FIRST_ORDINAL].toString())), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
Review Comment:
You are right, I missed that.
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(attributeName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(attributeName);
+ } else {
+ return handleEqual(attributeName, value);
+ }
+ }
+
+ case IN:
+ return in(
+ toColumnName(leftChild(predicate)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Expression notIn =
+ notIn(
+ toColumnName(leftChild(childFilter)),
+ Arrays.stream(childFilter.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(toColumnName(leftChild(childFilter))), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ return startsWith(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)).toString());
+ }
+ }
+
+ return null;
+ }
+
+ private static Object convertUTF8StringIfNecessary(Literal literal) {
+ if (literal.value() instanceof UTF8String) {
+ return ((UTF8String) literal.value()).toString();
+ }
+ return literal.value();
+ }
+
+ private static Expression handleEqual(String attribute, Object value) {
+ if (NaNUtil.isNaN(value)) {
+ return isNaN(attribute);
+ } else {
+ return equal(attribute, value);
+ }
+ }
+
+ private static boolean hasNoInFilter(Predicate predicate) {
+ Operation op = FILTERS.get(predicate.name());
+
+ if (op != null) {
+ switch (op) {
+ case AND:
+ And andPredicate = (And) predicate;
+ return hasNoInFilter(andPredicate.left()) && hasNoInFilter(andPredicate.right());
+ case OR:
+ Or orPredicate = (Or) predicate;
+ return hasNoInFilter(orPredicate.left()) && hasNoInFilter(orPredicate.right());
+ case NOT:
+ Not notPredicate = (Not) predicate;
+ return hasNoInFilter(notPredicate.child());
+ case IN:
+ return false;
+ default:
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ @SuppressWarnings("checkstyle:CyclomaticComplexity")
+ private static boolean valid(Predicate predicate) {
Review Comment:
I kept jumping from `convert` to `valid` when looking at the code. I feel having both the validation and conversion in one function would be a bit easier.
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(attributeName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(attributeName);
+ } else {
+ return handleEqual(attributeName, value);
+ }
+ }
+
+ case IN:
+ return in(
+ toColumnName(leftChild(predicate)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Expression notIn =
+ notIn(
+ toColumnName(leftChild(childFilter)),
+ Arrays.stream(childFilter.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(toColumnName(leftChild(childFilter))), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ return startsWith(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)).toString());
+ }
+ }
+
+ return null;
+ }
+
+ private static Object convertUTF8StringIfNecessary(Literal literal) {
+ if (literal.value() instanceof UTF8String) {
+ return ((UTF8String) literal.value()).toString();
+ }
+ return literal.value();
+ }
+
+ private static Expression handleEqual(String attribute, Object value) {
+ if (NaNUtil.isNaN(value)) {
+ return isNaN(attribute);
+ } else {
+ return equal(attribute, value);
+ }
+ }
+
+ private static boolean hasNoInFilter(Predicate predicate) {
+ Operation op = FILTERS.get(predicate.name());
+
+ if (op != null) {
+ switch (op) {
+ case AND:
+ And andPredicate = (And) predicate;
+ return hasNoInFilter(andPredicate.left()) && hasNoInFilter(andPredicate.right());
+ case OR:
+ Or orPredicate = (Or) predicate;
+ return hasNoInFilter(orPredicate.left()) && hasNoInFilter(orPredicate.right());
+ case NOT:
+ Not notPredicate = (Not) predicate;
+ return hasNoInFilter(notPredicate.child());
+ case IN:
+ return false;
+ default:
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ @SuppressWarnings("checkstyle:CyclomaticComplexity")
+ private static boolean valid(Predicate predicate) {
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case IS_NULL:
+ case NOT_NULL:
+ return child(predicate) instanceof NamedReference;
+
+ case LT:
+ case LT_EQ:
+ case GT:
+ case GT_EQ:
+ case EQ:
+ if (predicate.children().length != 2) {
+ return false;
+ }
+ return namedRefCompLit(predicate) || litCompNamedRef(predicate);
+
+ case IN:
+ if (!(leftChild(predicate) instanceof NamedReference)) {
+ return false;
+ } else {
+ return Arrays.stream(predicate.children())
+ .skip(1)
+ .allMatch(val -> val instanceof Literal);
+ }
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ return valid(notFilter.child());
+
+ case AND:
+ And andFilter = (And) predicate;
+ return valid(andFilter.left()) && valid(andFilter.right());
+
+ case OR:
+ Or orFilter = (Or) predicate;
+ return valid(orFilter.left()) && valid(orFilter.right());
+
+ case STARTS_WITH:
+ if (predicate.children().length != 2) {
+ return false;
+ }
+ return namedRefCompLit(predicate)
+ && ((Literal<?>) rightChild(predicate)).value() instanceof String;
+
+ case TRUE:
+ case FALSE:
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ private static boolean namedRefCompLit(Predicate predicate) {
Review Comment:
I think splitting this into `isRef` and `isLiteral` should allow us to combine the validation and conversion into one pass. Then we won't have to jump between methods when looking at the code.
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put(TRUE, Expression.Operation.TRUE)
+ .put(FALSE, Expression.Operation.FALSE)
+ .put(EQ, Expression.Operation.EQ)
+ .put(EQ_NULL_SAFE, Expression.Operation.EQ)
+ .put(GT, Expression.Operation.GT)
+ .put(GT_EQ, Expression.Operation.GT_EQ)
+ .put(LT, Expression.Operation.LT)
+ .put(LT_EQ, Expression.Operation.LT_EQ)
+ .put(IN, Expression.Operation.IN)
+ .put(IS_NULL, Expression.Operation.IS_NULL)
+ .put(NOT_NULL, Expression.Operation.NOT_NULL)
+ .put(AND, Expression.Operation.AND)
+ .put(OR, Expression.Operation.OR)
+ .put(NOT, Expression.Operation.NOT)
+ .put(STARTS_WITH, Expression.Operation.STARTS_WITH)
+ .build();
+
+ private static final int FIRST_ORDINAL = 0;
+ private static final int SECOND_ORDINAL = 1;
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
+
+ case LT:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[SECOND_ORDINAL]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ attributeName = predicate.children()[FIRST_ORDINAL].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = predicate.children()[SECOND_ORDINAL].toString();
+ value =
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value());
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
Review Comment:
It is unexpected and I think we wrap the conversion logic into try-catch, so we should be fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r954373839
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(attributeName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(attributeName);
+ } else {
+ return handleEqual(attributeName, value);
+ }
+ }
+
+ case IN:
+ return in(
+ toColumnName(leftChild(predicate)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Expression notIn =
+ notIn(
+ toColumnName(leftChild(childFilter)),
+ Arrays.stream(childFilter.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(toColumnName(leftChild(childFilter))), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ return startsWith(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)).toString());
+ }
+ }
+
+ return null;
+ }
+
+ private static Object convertUTF8StringIfNecessary(Literal literal) {
+ if (literal.value() instanceof UTF8String) {
+ return ((UTF8String) literal.value()).toString();
+ }
+ return literal.value();
+ }
+
+ private static Expression handleEqual(String attribute, Object value) {
+ if (NaNUtil.isNaN(value)) {
+ return isNaN(attribute);
+ } else {
+ return equal(attribute, value);
+ }
+ }
+
+ private static boolean hasNoInFilter(Predicate predicate) {
+ Operation op = FILTERS.get(predicate.name());
+
+ if (op != null) {
+ switch (op) {
+ case AND:
+ And andPredicate = (And) predicate;
+ return hasNoInFilter(andPredicate.left()) && hasNoInFilter(andPredicate.right());
+ case OR:
+ Or orPredicate = (Or) predicate;
+ return hasNoInFilter(orPredicate.left()) && hasNoInFilter(orPredicate.right());
+ case NOT:
+ Not notPredicate = (Not) predicate;
+ return hasNoInFilter(notPredicate.child());
+ case IN:
+ return false;
+ default:
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ @SuppressWarnings("checkstyle:CyclomaticComplexity")
+ private static boolean valid(Predicate predicate) {
Review Comment:
Agree!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#issuecomment-1188401258
cc @rdblue @aokolnychyi @RussellSpitzer
Could you please take a look when you have time? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r950615128
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put(TRUE, Expression.Operation.TRUE)
+ .put(FALSE, Expression.Operation.FALSE)
+ .put(EQ, Expression.Operation.EQ)
+ .put(EQ_NULL_SAFE, Expression.Operation.EQ)
+ .put(GT, Expression.Operation.GT)
+ .put(GT_EQ, Expression.Operation.GT_EQ)
+ .put(LT, Expression.Operation.LT)
+ .put(LT_EQ, Expression.Operation.LT_EQ)
+ .put(IN, Expression.Operation.IN)
+ .put(IS_NULL, Expression.Operation.IS_NULL)
+ .put(NOT_NULL, Expression.Operation.NOT_NULL)
+ .put(AND, Expression.Operation.AND)
+ .put(OR, Expression.Operation.OR)
+ .put(NOT, Expression.Operation.NOT)
+ .put(STARTS_WITH, Expression.Operation.STARTS_WITH)
+ .build();
+
+ private static final int FIRST_ORDINAL = 0;
+ private static final int SECOND_ORDINAL = 1;
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
Review Comment:
Weird. I think my original comment wasn't posted. Let me double it below.
I was saying that we shouldn't use unquoting workarounds we had for V1 filters. Instead, we should use field arrays in V2 predicates.
I'd consider something like this (needs to be polished further).
```
private static final Joiner DOT = Joiner.on(".");
...
@SuppressWarnings("unchecked")
private static <T> T child(Predicate predicate) {
org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
Preconditions.checkArgument(children.length == 1, "TBD");
return (T) children[0];
}
@SuppressWarnings("unchecked")
private static <T> T leftChild(Predicate predicate) {
return (T) ...
}
@SuppressWarnings("unchecked")
private static <T> T rightChild(Predicate predicate) {
return (T) ...
}
private static String toColumnName(NamedReference ref) {
return DOT.join(ref.fieldNames());
}
```
And using it like this:
```
...
case IS_NULL:
return isNull(toColumnName(child(predicate)));
case NOT_NULL:
return notNull(toColumnName(child(predicate)));
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#issuecomment-1219052390
Thanks a lot @kbendick for taking time reviewing the PR! I have addressed the comments. Could you please take one more look when you are free? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r950635704
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put(TRUE, Expression.Operation.TRUE)
+ .put(FALSE, Expression.Operation.FALSE)
+ .put(EQ, Expression.Operation.EQ)
+ .put(EQ_NULL_SAFE, Expression.Operation.EQ)
+ .put(GT, Expression.Operation.GT)
+ .put(GT_EQ, Expression.Operation.GT_EQ)
+ .put(LT, Expression.Operation.LT)
+ .put(LT_EQ, Expression.Operation.LT_EQ)
+ .put(IN, Expression.Operation.IN)
+ .put(IS_NULL, Expression.Operation.IS_NULL)
+ .put(NOT_NULL, Expression.Operation.NOT_NULL)
+ .put(AND, Expression.Operation.AND)
+ .put(OR, Expression.Operation.OR)
+ .put(NOT, Expression.Operation.NOT)
+ .put(STARTS_WITH, Expression.Operation.STARTS_WITH)
+ .build();
+
+ private static final int FIRST_ORDINAL = 0;
+ private static final int SECOND_ORDINAL = 1;
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
Review Comment:
Mainly trying to validate the Spark filter logic. In binary comparison predicate, Expression cmp Expression is also valid in Spark, but I think Iceberg only accepts Attribute cmp Literal, so I also make sure it's either attribute cmp lit or lit cmp attribute.
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put(TRUE, Expression.Operation.TRUE)
+ .put(FALSE, Expression.Operation.FALSE)
+ .put(EQ, Expression.Operation.EQ)
+ .put(EQ_NULL_SAFE, Expression.Operation.EQ)
+ .put(GT, Expression.Operation.GT)
+ .put(GT_EQ, Expression.Operation.GT_EQ)
+ .put(LT, Expression.Operation.LT)
+ .put(LT_EQ, Expression.Operation.LT_EQ)
+ .put(IN, Expression.Operation.IN)
+ .put(IS_NULL, Expression.Operation.IS_NULL)
+ .put(NOT_NULL, Expression.Operation.NOT_NULL)
+ .put(AND, Expression.Operation.AND)
+ .put(OR, Expression.Operation.OR)
+ .put(NOT, Expression.Operation.NOT)
+ .put(STARTS_WITH, Expression.Operation.STARTS_WITH)
+ .build();
+
+ private static final int FIRST_ORDINAL = 0;
+ private static final int SECOND_ORDINAL = 1;
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
+
+ case LT:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[SECOND_ORDINAL]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ attributeName = predicate.children()[FIRST_ORDINAL].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = predicate.children()[SECOND_ORDINAL].toString();
+ value =
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value());
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(unquote(attributeName), value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(unquote(attributeName));
+ } else {
+ return handleEqual(unquote(attributeName), value);
+ }
+ }
+
+ case IN:
+ return in(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((LiteralValue) val).value()))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Expression notIn =
+ notIn(
+ unquote(childFilter.children()[FIRST_ORDINAL].toString()),
+ Arrays.stream(childFilter.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((LiteralValue) val).value()))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(unquote(childFilter.children()[FIRST_ORDINAL].toString())), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
Review Comment:
This is required. Otherwise, I can't use the same variable names (`left` and `right`) in difference case blocks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r950635741
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put(TRUE, Expression.Operation.TRUE)
+ .put(FALSE, Expression.Operation.FALSE)
+ .put(EQ, Expression.Operation.EQ)
+ .put(EQ_NULL_SAFE, Expression.Operation.EQ)
+ .put(GT, Expression.Operation.GT)
+ .put(GT_EQ, Expression.Operation.GT_EQ)
+ .put(LT, Expression.Operation.LT)
+ .put(LT_EQ, Expression.Operation.LT_EQ)
+ .put(IN, Expression.Operation.IN)
+ .put(IS_NULL, Expression.Operation.IS_NULL)
+ .put(NOT_NULL, Expression.Operation.NOT_NULL)
+ .put(AND, Expression.Operation.AND)
+ .put(OR, Expression.Operation.OR)
+ .put(NOT, Expression.Operation.NOT)
+ .put(STARTS_WITH, Expression.Operation.STARTS_WITH)
+ .build();
+
+ private static final int FIRST_ORDINAL = 0;
+ private static final int SECOND_ORDINAL = 1;
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
+
+ case LT:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
Review Comment:
Changed to `Literal`. Thanks!
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put(TRUE, Expression.Operation.TRUE)
+ .put(FALSE, Expression.Operation.FALSE)
+ .put(EQ, Expression.Operation.EQ)
+ .put(EQ_NULL_SAFE, Expression.Operation.EQ)
+ .put(GT, Expression.Operation.GT)
+ .put(GT_EQ, Expression.Operation.GT_EQ)
+ .put(LT, Expression.Operation.LT)
+ .put(LT_EQ, Expression.Operation.LT_EQ)
+ .put(IN, Expression.Operation.IN)
+ .put(IS_NULL, Expression.Operation.IS_NULL)
+ .put(NOT_NULL, Expression.Operation.NOT_NULL)
+ .put(AND, Expression.Operation.AND)
+ .put(OR, Expression.Operation.OR)
+ .put(NOT, Expression.Operation.NOT)
+ .put(STARTS_WITH, Expression.Operation.STARTS_WITH)
+ .build();
+
+ private static final int FIRST_ORDINAL = 0;
+ private static final int SECOND_ORDINAL = 1;
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
+
+ case LT:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[SECOND_ORDINAL]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ attributeName = predicate.children()[FIRST_ORDINAL].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = predicate.children()[SECOND_ORDINAL].toString();
+ value =
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value());
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(unquote(attributeName), value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(unquote(attributeName));
+ } else {
+ return handleEqual(unquote(attributeName), value);
+ }
+ }
+
+ case IN:
+ return in(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((LiteralValue) val).value()))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Expression notIn =
+ notIn(
+ unquote(childFilter.children()[FIRST_ORDINAL].toString()),
+ Arrays.stream(childFilter.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((LiteralValue) val).value()))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(unquote(childFilter.children()[FIRST_ORDINAL].toString())), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ {
+ return startsWith(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[SECOND_ORDINAL]).value())
+ .toString());
+ }
+ }
+ }
+
+ return null;
+ }
+
+ private static Object convertUTF8StringIfNecessary(Object value) {
Review Comment:
Fixed. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r946166978
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (checkIfPredicateValid(predicate) == null) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[0].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[0].toString()));
+
+ case LT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value = null;
+ String attributeName = "";
+ if (predicate.children()[1] instanceof LiteralValue) {
+ attributeName = predicate.children()[0].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = predicate.children()[1].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value());
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(unquote(attributeName), value);
+ } else if (predicate.name().equals("<=>")) {
+ if (value == null) {
+ return isNull(unquote(attributeName));
+ } else {
+ return handleEqual(unquote(attributeName), value);
+ }
+ }
+ break;
+
+ case IN:
+ Object[] inValues = new Object[predicate.children().length - 1];
+ for (int i = 1; i < predicate.children().length; i++) {
+ inValues[i - 1] =
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[i]).value());
+ }
+ return in(
+ unquote(predicate.children()[0].toString()),
+ Stream.of(inValues).filter(Objects::nonNull).collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Object[] notInValues = new Object[childFilter.children().length - 1];
+ for (int i = 1; i < childFilter.children().length; i++) {
+ notInValues[i - 1] =
+ convertUTF8StringIfNecessary(((LiteralValue) childFilter.children()[i]).value());
+ }
+ Expression notIn =
+ notIn(
+ unquote(childFilter.children()[0].toString()),
+ Stream.of(notInValues).collect(Collectors.toList()));
+ return and(notNull(unquote(childFilter.children()[0].toString())), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ {
+ return startsWith(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value())
+ .toString());
+ }
+ }
+ }
+
+ return null;
+ }
+
+ private static Object convertUTF8StringIfNecessary(Object value) {
+ if (value instanceof UTF8String) {
+ return ((UTF8String) value).toString();
+ }
+ return value;
+ }
+
+ private static Expression handleEqual(String attribute, Object value) {
+ if (NaNUtil.isNaN(value)) {
+ return isNaN(attribute);
+ } else {
+ return equal(attribute, value);
+ }
+ }
+
+ private static String unquote(String attributeName) {
+ Matcher matcher = BACKTICKS_PATTERN.matcher(attributeName);
+ return matcher.replaceAll("$2");
+ }
+
+ private static boolean hasNoInFilter(Predicate predicate) {
+ Expression.Operation op = FILTERS.get(predicate.name());
+
+ if (op != null) {
+ switch (op) {
+ case AND:
+ And andPredicate = (And) predicate;
+ return hasNoInFilter(andPredicate.left()) && hasNoInFilter(andPredicate.right());
+ case OR:
+ Or orPredicate = (Or) predicate;
+ return hasNoInFilter(orPredicate.left()) && hasNoInFilter(orPredicate.right());
+ case NOT:
+ Not notPredicate = (Not) predicate;
+ return hasNoInFilter(notPredicate.child());
+ case IN:
+ return false;
+ default:
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ @SuppressWarnings("checkstyle:CyclomaticComplexity")
+ private static Predicate checkIfPredicateValid(Predicate predicate) {
Review Comment:
I think we usually prefer shorter method names, so maybe valid(Predicate predicate) is better? Also, kind of related to Russell's comment, why cant the return value be a boolean instead of predicate? (valid seems to be a true/false question)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
kbendick commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r948130071
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[0].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[0].toString()));
+
+ case LT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
Review Comment:
Nit / non-blocking style question: Would it make sense to use named constant for the ordinals? They're used quite a lot.
Maybe the following?
```
private static int FIRST_PREDICATE_CHILD_ORDINAL = 0;
private static int SECOND_PREDICATE_CHILD_ORGINAL = 1;
```
Not tied to the naming but Anton suggested I do similar in the `truncate` v2 function PR and it turned out nicely. Might be overkill since they don't correspond to much besides first and second though.
https://github.com/apache/iceberg/blob/b3062e443848c9badd7332c4b26c14f2b257cc4a/spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/functions/TruncateFunction.java#L56-L57
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
kbendick commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r948140194
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/TestSparkV2Filters.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.temporal.ChronoUnit;
+import java.util.Map;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DateType$;
+import org.apache.spark.sql.types.IntegerType$;
+import org.apache.spark.sql.types.TimestampType$;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestSparkV2Filters {
Review Comment:
Nit / non-blocking / for follow up if desired: We might break these up into two or three sets of tests.
I can see:
1. Literal to Attribute
2. Attribute to Literal
3. Invalid assertions.
This could totally be done in a follow up PR, but because the stringly typed nature of these areas is somewhat error prone in my opinion I think we should try to make the tests extensible. But not important for this PR imo (which has been around long enough and would be good to be using).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r954373647
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(attributeName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(attributeName);
+ } else {
+ return handleEqual(attributeName, value);
+ }
+ }
+
+ case IN:
+ return in(
+ toColumnName(leftChild(predicate)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
Review Comment:
Fixed. Thanks!
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(attributeName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(attributeName);
+ } else {
+ return handleEqual(attributeName, value);
+ }
+ }
+
+ case IN:
+ return in(
+ toColumnName(leftChild(predicate)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Expression notIn =
+ notIn(
+ toColumnName(leftChild(childFilter)),
+ Arrays.stream(childFilter.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
Review Comment:
Fixed. Thanks!
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(attributeName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(attributeName);
+ } else {
+ return handleEqual(attributeName, value);
+ }
+ }
+
+ case IN:
+ return in(
+ toColumnName(leftChild(predicate)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Expression notIn =
+ notIn(
+ toColumnName(leftChild(childFilter)),
+ Arrays.stream(childFilter.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(toColumnName(leftChild(childFilter))), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ return startsWith(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)).toString());
Review Comment:
Seems this `toString` is needed.
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(attributeName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(attributeName);
+ } else {
+ return handleEqual(attributeName, value);
+ }
+ }
+
+ case IN:
+ return in(
+ toColumnName(leftChild(predicate)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Expression notIn =
+ notIn(
+ toColumnName(leftChild(childFilter)),
+ Arrays.stream(childFilter.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(toColumnName(leftChild(childFilter))), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ return startsWith(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)).toString());
+ }
+ }
+
+ return null;
+ }
+
+ private static Object convertUTF8StringIfNecessary(Literal literal) {
+ if (literal.value() instanceof UTF8String) {
+ return ((UTF8String) literal.value()).toString();
+ }
+ return literal.value();
+ }
+
+ private static Expression handleEqual(String attribute, Object value) {
+ if (NaNUtil.isNaN(value)) {
+ return isNaN(attribute);
+ } else {
+ return equal(attribute, value);
+ }
+ }
+
+ private static boolean hasNoInFilter(Predicate predicate) {
+ Operation op = FILTERS.get(predicate.name());
+
+ if (op != null) {
+ switch (op) {
+ case AND:
+ And andPredicate = (And) predicate;
+ return hasNoInFilter(andPredicate.left()) && hasNoInFilter(andPredicate.right());
+ case OR:
+ Or orPredicate = (Or) predicate;
+ return hasNoInFilter(orPredicate.left()) && hasNoInFilter(orPredicate.right());
+ case NOT:
+ Not notPredicate = (Not) predicate;
+ return hasNoInFilter(notPredicate.child());
+ case IN:
+ return false;
+ default:
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ @SuppressWarnings("checkstyle:CyclomaticComplexity")
+ private static boolean valid(Predicate predicate) {
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case IS_NULL:
+ case NOT_NULL:
+ return child(predicate) instanceof NamedReference;
+
+ case LT:
+ case LT_EQ:
+ case GT:
+ case GT_EQ:
+ case EQ:
+ if (predicate.children().length != 2) {
+ return false;
+ }
+ return namedRefCompLit(predicate) || litCompNamedRef(predicate);
+
+ case IN:
+ if (!(leftChild(predicate) instanceof NamedReference)) {
+ return false;
+ } else {
+ return Arrays.stream(predicate.children())
+ .skip(1)
+ .allMatch(val -> val instanceof Literal);
+ }
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ return valid(notFilter.child());
+
+ case AND:
+ And andFilter = (And) predicate;
+ return valid(andFilter.left()) && valid(andFilter.right());
+
+ case OR:
+ Or orFilter = (Or) predicate;
+ return valid(orFilter.left()) && valid(orFilter.right());
+
+ case STARTS_WITH:
+ if (predicate.children().length != 2) {
+ return false;
+ }
+ return namedRefCompLit(predicate)
+ && ((Literal<?>) rightChild(predicate)).value() instanceof String;
+
+ case TRUE:
+ case FALSE:
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ private static boolean namedRefCompLit(Predicate predicate) {
Review Comment:
Agree!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#issuecomment-1203076657
gentle ping @rdblue @aokolnychyi @RussellSpitzer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#issuecomment-1216822184
@szehon-ho Thank you very much for taking time to review. I have addressed the comments. Could you please take another look when you are free? Thanks a lot!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r950617908
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put(TRUE, Expression.Operation.TRUE)
+ .put(FALSE, Expression.Operation.FALSE)
+ .put(EQ, Expression.Operation.EQ)
+ .put(EQ_NULL_SAFE, Expression.Operation.EQ)
+ .put(GT, Expression.Operation.GT)
+ .put(GT_EQ, Expression.Operation.GT_EQ)
+ .put(LT, Expression.Operation.LT)
+ .put(LT_EQ, Expression.Operation.LT_EQ)
+ .put(IN, Expression.Operation.IN)
+ .put(IS_NULL, Expression.Operation.IS_NULL)
+ .put(NOT_NULL, Expression.Operation.NOT_NULL)
+ .put(AND, Expression.Operation.AND)
+ .put(OR, Expression.Operation.OR)
+ .put(NOT, Expression.Operation.NOT)
+ .put(STARTS_WITH, Expression.Operation.STARTS_WITH)
+ .build();
+
+ private static final int FIRST_ORDINAL = 0;
+ private static final int SECOND_ORDINAL = 1;
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
+
+ case LT:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
Review Comment:
Isn't `LiteralValue` private to Spark? Should we use the public interface?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r948660012
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/TestSparkV2Filters.java:
##########
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.temporal.ChronoUnit;
+import java.util.Map;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DateType$;
+import org.apache.spark.sql.types.IntegerType$;
+import org.apache.spark.sql.types.TimestampType$;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestSparkV2Filters {
+
+ @Test
+ public void testV2Filters() {
+ Map<String, String> attrMap = Maps.newHashMap();
+ attrMap.put("id", "id");
+ attrMap.put("`i.d`", "i.d");
+ attrMap.put("`i``d`", "i`d");
+ attrMap.put("`d`.b.`dd```", "d.b.dd`");
+ attrMap.put("a.`aa```.c", "a.aa`.c");
+
+ attrMap.forEach(
+ (quoted, unquoted) -> {
+ NamedReference namedReference = FieldReference.apply(quoted);
+ org.apache.spark.sql.connector.expressions.Expression[] attrOnly =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference};
+
+ LiteralValue value = new LiteralValue(1, IntegerType$.MODULE$);
+ org.apache.spark.sql.connector.expressions.Expression[] attrAndValue =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference, value};
+ org.apache.spark.sql.connector.expressions.Expression[] valueAndAttr =
+ new org.apache.spark.sql.connector.expressions.Expression[] {value, namedReference};
+
+ Predicate isNull = new Predicate("IS_NULL", attrOnly);
+ Expression expectedIsNull = Expressions.isNull(unquoted);
+ Expression actualIsNull = SparkV2Filters.convert(isNull);
+ Assert.assertEquals(
+ "IsNull must match", expectedIsNull.toString(), actualIsNull.toString());
+
+ Predicate isNotNull = new Predicate("IS_NOT_NULL", attrOnly);
+ Expression expectedIsNotNull = Expressions.notNull(unquoted);
+ Expression actualIsNotNull = SparkV2Filters.convert(isNotNull);
+ Assert.assertEquals(
+ "IsNotNull must match", expectedIsNotNull.toString(), actualIsNotNull.toString());
+
+ Predicate lt1 = new Predicate("<", attrAndValue);
+ Expression expectedLt1 = Expressions.lessThan(unquoted, 1);
+ Expression actualLt1 = SparkV2Filters.convert(lt1);
+ Assert.assertEquals("LessThan must match", expectedLt1.toString(), actualLt1.toString());
+
+ Predicate lt2 = new Predicate("<", valueAndAttr);
+ Expression expectedLt2 = Expressions.greaterThan(unquoted, 1);
+ Expression actualLt2 = SparkV2Filters.convert(lt2);
+ Assert.assertEquals("LessThan must match", expectedLt2.toString(), actualLt2.toString());
+
+ Predicate ltEq1 = new Predicate("<=", attrAndValue);
+ Expression expectedLtEq1 = Expressions.lessThanOrEqual(unquoted, 1);
+ Expression actualLtEq1 = SparkV2Filters.convert(ltEq1);
+ Assert.assertEquals(
+ "LessThanOrEqual must match", expectedLtEq1.toString(), actualLtEq1.toString());
+
+ Predicate ltEq2 = new Predicate("<=", valueAndAttr);
+ Expression expectedLtEq2 = Expressions.greaterThanOrEqual(unquoted, 1);
+ Expression actualLtEq2 = SparkV2Filters.convert(ltEq2);
+ Assert.assertEquals(
+ "LessThanOrEqual must match", expectedLtEq2.toString(), actualLtEq2.toString());
+
+ Predicate gt1 = new Predicate(">", attrAndValue);
+ Expression expectedGt1 = Expressions.greaterThan(unquoted, 1);
+ Expression actualGt1 = SparkV2Filters.convert(gt1);
+ Assert.assertEquals(
+ "GreaterThan must match", expectedGt1.toString(), actualGt1.toString());
+
+ Predicate gt2 = new Predicate(">", valueAndAttr);
+ Expression expectedGt2 = Expressions.lessThan(unquoted, 1);
+ Expression actualGt2 = SparkV2Filters.convert(gt2);
+ Assert.assertEquals(
+ "GreaterThan must match", expectedGt2.toString(), actualGt2.toString());
+
+ Predicate gtEq1 = new Predicate(">=", attrAndValue);
+ Expression expectedGtEq1 = Expressions.greaterThanOrEqual(unquoted, 1);
+ Expression actualGtEq1 = SparkV2Filters.convert(gtEq1);
+ Assert.assertEquals(
+ "GreaterThanOrEqual must match", expectedGtEq1.toString(), actualGtEq1.toString());
+
+ Predicate gtEq2 = new Predicate(">=", valueAndAttr);
+ Expression expectedGtEq2 = Expressions.lessThanOrEqual(unquoted, 1);
+ Expression actualGtEq2 = SparkV2Filters.convert(gtEq2);
+ Assert.assertEquals(
+ "GreaterThanOrEqual must match", expectedGtEq2.toString(), actualGtEq2.toString());
+
+ Predicate eq1 = new Predicate("=", attrAndValue);
+ Expression expectedEq1 = Expressions.equal(unquoted, 1);
+ Expression actualEq1 = SparkV2Filters.convert(eq1);
+ Assert.assertEquals("EqualTo must match", expectedEq1.toString(), actualEq1.toString());
+
+ Predicate eq2 = new Predicate("=", valueAndAttr);
+ Expression expectedEq2 = Expressions.equal(unquoted, 1);
+ Expression actualEq2 = SparkV2Filters.convert(eq2);
+ Assert.assertEquals("EqualTo must match", expectedEq2.toString(), actualEq2.toString());
+
+ Predicate eqNullSafe1 = new Predicate("<=>", attrAndValue);
+ Expression expectedEqNullSafe1 = Expressions.equal(unquoted, 1);
+ Expression actualEqNullSafe1 = SparkV2Filters.convert(eqNullSafe1);
+ Assert.assertEquals(
+ "EqualNullSafe must match",
+ expectedEqNullSafe1.toString(),
+ actualEqNullSafe1.toString());
+
+ Predicate eqNullSafe2 = new Predicate("<=>", valueAndAttr);
+ Expression expectedEqNullSafe2 = Expressions.equal(unquoted, 1);
+ Expression actualEqNullSafe2 = SparkV2Filters.convert(eqNullSafe2);
+ Assert.assertEquals(
+ "EqualNullSafe must match",
+ expectedEqNullSafe2.toString(),
+ actualEqNullSafe2.toString());
+
+ Predicate in = new Predicate("IN", attrAndValue);
+ Expression expectedIn = Expressions.in(unquoted, 1);
+ Expression actualIn = SparkV2Filters.convert(in);
+ Assert.assertEquals("In must match", expectedIn.toString(), actualIn.toString());
+
+ Predicate and = new And(lt1, eq1);
+ Expression expectedAnd = Expressions.and(expectedLt1, expectedEq1);
+ Expression actualAnd = SparkV2Filters.convert(and);
+ Assert.assertEquals("And must match", expectedAnd.toString(), actualAnd.toString());
+
+ Predicate invalid = new Predicate("<", attrOnly);
+ Predicate andWithInvalidLeft = new And(invalid, eq1);
+ Expression convertedAnd = SparkV2Filters.convert(andWithInvalidLeft);
+ Assert.assertEquals("And must match", convertedAnd, null);
+
+ Predicate or = new Or(lt1, eq1);
+ Expression expectedOr = Expressions.or(expectedLt1, expectedEq1);
+ Expression actualOr = SparkV2Filters.convert(or);
+ Assert.assertEquals("Or must match", expectedOr.toString(), actualOr.toString());
+
+ Predicate orWithInvalidLeft = new Or(invalid, eq1);
+ Expression convertedOr = SparkV2Filters.convert(orWithInvalidLeft);
+ Assert.assertEquals("Or must match", convertedOr, null);
+
+ Predicate not = new Not(lt1);
+ Expression expectedNot = Expressions.not(expectedLt1);
+ Expression actualNot = SparkV2Filters.convert(not);
+ Assert.assertEquals("Not must match", expectedNot.toString(), actualNot.toString());
+ });
+ }
+
+ @Test
+ public void testTimestampFilterConversion() {
+ Instant instant = Instant.parse("2018-10-18T00:00:57.907Z");
+ long epochMicros = ChronoUnit.MICROS.between(Instant.EPOCH, instant);
+
+ NamedReference namedReference = FieldReference.apply("x");
+ LiteralValue ts = new LiteralValue(epochMicros, TimestampType$.MODULE$);
+ org.apache.spark.sql.connector.expressions.Expression[] attrAndValue =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference, ts};
+
+ Predicate predicate = new Predicate(">", attrAndValue);
+ Expression tsExpression = SparkV2Filters.convert(predicate);
+ Expression rawExpression = Expressions.greaterThan("x", epochMicros);
+
+ Assert.assertEquals(
+ "Generated Timestamp expression should be correct",
+ rawExpression.toString(),
+ tsExpression.toString());
+ }
+
+ @Test
+ public void testDateFilterConversion() {
+ LocalDate localDate = LocalDate.parse("2018-10-18");
+ long epochDay = localDate.toEpochDay();
+
+ NamedReference namedReference = FieldReference.apply("x");
+ LiteralValue ts = new LiteralValue(epochDay, DateType$.MODULE$);
+ org.apache.spark.sql.connector.expressions.Expression[] attrAndValue =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference, ts};
+
+ Predicate predicate = new Predicate(">", attrAndValue);
+ Expression dateExpression = SparkV2Filters.convert(predicate);
+ Expression rawExpression = Expressions.greaterThan("x", epochDay);
+
+ Assert.assertEquals(
+ "Generated date expression should be correct",
+ rawExpression.toString(),
+ dateExpression.toString());
+ }
+
+ @Test
+ public void testNestedInInsideNot() {
+ NamedReference namedReference1 = FieldReference.apply("col1");
+ LiteralValue v1 = new LiteralValue(1, IntegerType$.MODULE$);
Review Comment:
Fixed. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
kbendick commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r950636390
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put(TRUE, Expression.Operation.TRUE)
+ .put(FALSE, Expression.Operation.FALSE)
+ .put(EQ, Expression.Operation.EQ)
+ .put(EQ_NULL_SAFE, Expression.Operation.EQ)
+ .put(GT, Expression.Operation.GT)
+ .put(GT_EQ, Expression.Operation.GT_EQ)
+ .put(LT, Expression.Operation.LT)
+ .put(LT_EQ, Expression.Operation.LT_EQ)
+ .put(IN, Expression.Operation.IN)
+ .put(IS_NULL, Expression.Operation.IS_NULL)
+ .put(NOT_NULL, Expression.Operation.NOT_NULL)
+ .put(AND, Expression.Operation.AND)
+ .put(OR, Expression.Operation.OR)
+ .put(NOT, Expression.Operation.NOT)
+ .put(STARTS_WITH, Expression.Operation.STARTS_WITH)
+ .build();
+
+ private static final int FIRST_ORDINAL = 0;
+ private static final int SECOND_ORDINAL = 1;
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
Review Comment:
Oh big +1 I like this a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r950643614
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(children.length == 1, "%s should have one child", predicate);
Review Comment:
Will fix. Thanks
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/TestSparkV2Filters.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.temporal.ChronoUnit;
+import java.util.Map;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DataTypes;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestSparkV2Filters {
+
+ @Test
+ public void testV2Filters() {
+ Map<String, String> attrMap = Maps.newHashMap();
+ attrMap.put("id", "id");
+ attrMap.put("`i.d`", "i.d");
+ attrMap.put("`i``d`", "i`d");
+ attrMap.put("`d`.b.`dd```", "d.b.dd`");
+ attrMap.put("a.`aa```.c", "a.aa`.c");
+
+ attrMap.forEach(
+ (quoted, unquoted) -> {
+ NamedReference namedReference = FieldReference.apply(quoted);
+ org.apache.spark.sql.connector.expressions.Expression[] attrOnly =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference};
+
+ LiteralValue value = new LiteralValue(1, DataTypes.IntegerType);
+ org.apache.spark.sql.connector.expressions.Expression[] attrAndValue =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference, value};
+ org.apache.spark.sql.connector.expressions.Expression[] valueAndAttr =
+ new org.apache.spark.sql.connector.expressions.Expression[] {value, namedReference};
+
+ Predicate isNull = new Predicate("IS_NULL", attrOnly);
+ Expression expectedIsNull = Expressions.isNull(unquoted);
+ Expression actualIsNull = SparkV2Filters.convert(isNull);
+ Assert.assertEquals(
+ "IsNull must match", expectedIsNull.toString(), actualIsNull.toString());
+
+ Predicate isNotNull = new Predicate("IS_NOT_NULL", attrOnly);
+ Expression expectedIsNotNull = Expressions.notNull(unquoted);
+ Expression actualIsNotNull = SparkV2Filters.convert(isNotNull);
+ Assert.assertEquals(
+ "IsNotNull must match", expectedIsNotNull.toString(), actualIsNotNull.toString());
+
+ Predicate lt1 = new Predicate("<", attrAndValue);
+ Expression expectedLt1 = Expressions.lessThan(unquoted, 1);
+ Expression actualLt1 = SparkV2Filters.convert(lt1);
+ Assert.assertEquals("LessThan must match", expectedLt1.toString(), actualLt1.toString());
+
+ Predicate lt2 = new Predicate("<", valueAndAttr);
+ Expression expectedLt2 = Expressions.greaterThan(unquoted, 1);
+ Expression actualLt2 = SparkV2Filters.convert(lt2);
+ Assert.assertEquals("LessThan must match", expectedLt2.toString(), actualLt2.toString());
+
+ Predicate ltEq1 = new Predicate("<=", attrAndValue);
+ Expression expectedLtEq1 = Expressions.lessThanOrEqual(unquoted, 1);
+ Expression actualLtEq1 = SparkV2Filters.convert(ltEq1);
+ Assert.assertEquals(
+ "LessThanOrEqual must match", expectedLtEq1.toString(), actualLtEq1.toString());
+
+ Predicate ltEq2 = new Predicate("<=", valueAndAttr);
+ Expression expectedLtEq2 = Expressions.greaterThanOrEqual(unquoted, 1);
+ Expression actualLtEq2 = SparkV2Filters.convert(ltEq2);
+ Assert.assertEquals(
+ "LessThanOrEqual must match", expectedLtEq2.toString(), actualLtEq2.toString());
+
+ Predicate gt1 = new Predicate(">", attrAndValue);
+ Expression expectedGt1 = Expressions.greaterThan(unquoted, 1);
+ Expression actualGt1 = SparkV2Filters.convert(gt1);
+ Assert.assertEquals(
+ "GreaterThan must match", expectedGt1.toString(), actualGt1.toString());
+
+ Predicate gt2 = new Predicate(">", valueAndAttr);
+ Expression expectedGt2 = Expressions.lessThan(unquoted, 1);
+ Expression actualGt2 = SparkV2Filters.convert(gt2);
+ Assert.assertEquals(
+ "GreaterThan must match", expectedGt2.toString(), actualGt2.toString());
+
+ Predicate gtEq1 = new Predicate(">=", attrAndValue);
+ Expression expectedGtEq1 = Expressions.greaterThanOrEqual(unquoted, 1);
+ Expression actualGtEq1 = SparkV2Filters.convert(gtEq1);
+ Assert.assertEquals(
+ "GreaterThanOrEqual must match", expectedGtEq1.toString(), actualGtEq1.toString());
+
+ Predicate gtEq2 = new Predicate(">=", valueAndAttr);
+ Expression expectedGtEq2 = Expressions.lessThanOrEqual(unquoted, 1);
+ Expression actualGtEq2 = SparkV2Filters.convert(gtEq2);
+ Assert.assertEquals(
+ "GreaterThanOrEqual must match", expectedGtEq2.toString(), actualGtEq2.toString());
+
+ Predicate eq1 = new Predicate("=", attrAndValue);
+ Expression expectedEq1 = Expressions.equal(unquoted, 1);
+ Expression actualEq1 = SparkV2Filters.convert(eq1);
+ Assert.assertEquals("EqualTo must match", expectedEq1.toString(), actualEq1.toString());
+
+ Predicate eq2 = new Predicate("=", valueAndAttr);
+ Expression expectedEq2 = Expressions.equal(unquoted, 1);
+ Expression actualEq2 = SparkV2Filters.convert(eq2);
+ Assert.assertEquals("EqualTo must match", expectedEq2.toString(), actualEq2.toString());
+
+ Predicate eqNullSafe1 = new Predicate("<=>", attrAndValue);
+ Expression expectedEqNullSafe1 = Expressions.equal(unquoted, 1);
+ Expression actualEqNullSafe1 = SparkV2Filters.convert(eqNullSafe1);
+ Assert.assertEquals(
+ "EqualNullSafe must match",
+ expectedEqNullSafe1.toString(),
+ actualEqNullSafe1.toString());
+
+ Predicate eqNullSafe2 = new Predicate("<=>", valueAndAttr);
+ Expression expectedEqNullSafe2 = Expressions.equal(unquoted, 1);
+ Expression actualEqNullSafe2 = SparkV2Filters.convert(eqNullSafe2);
+ Assert.assertEquals(
+ "EqualNullSafe must match",
+ expectedEqNullSafe2.toString(),
+ actualEqNullSafe2.toString());
+
+ Predicate in = new Predicate("IN", attrAndValue);
+ Expression expectedIn = Expressions.in(unquoted, 1);
+ Expression actualIn = SparkV2Filters.convert(in);
+ Assert.assertEquals("In must match", expectedIn.toString(), actualIn.toString());
+
+ Predicate and = new And(lt1, eq1);
+ Expression expectedAnd = Expressions.and(expectedLt1, expectedEq1);
+ Expression actualAnd = SparkV2Filters.convert(and);
+ Assert.assertEquals("And must match", expectedAnd.toString(), actualAnd.toString());
+
+ Predicate invalid = new Predicate("<", attrOnly);
+ Predicate andWithInvalidLeft = new And(invalid, eq1);
+ Expression convertedAnd = SparkV2Filters.convert(andWithInvalidLeft);
+ Assert.assertEquals("And must match", convertedAnd, null);
+
+ Predicate or = new Or(lt1, eq1);
+ Expression expectedOr = Expressions.or(expectedLt1, expectedEq1);
+ Expression actualOr = SparkV2Filters.convert(or);
+ Assert.assertEquals("Or must match", expectedOr.toString(), actualOr.toString());
+
+ Predicate orWithInvalidLeft = new Or(invalid, eq1);
+ Expression convertedOr = SparkV2Filters.convert(orWithInvalidLeft);
+ Assert.assertEquals("Or must match", convertedOr, null);
+
+ Predicate not = new Not(lt1);
+ Expression expectedNot = Expressions.not(expectedLt1);
+ Expression actualNot = SparkV2Filters.convert(not);
+ Assert.assertEquals("Not must match", expectedNot.toString(), actualNot.toString());
+ });
+ }
+
+ @Test
+ public void testTimestampFilterConversion() {
+ Instant instant = Instant.parse("2018-10-18T00:00:57.907Z");
+ long epochMicros = ChronoUnit.MICROS.between(Instant.EPOCH, instant);
+
+ NamedReference namedReference = FieldReference.apply("x");
+ LiteralValue ts = new LiteralValue(epochMicros, DataTypes.TimestampType);
+ org.apache.spark.sql.connector.expressions.Expression[] attrAndValue =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference, ts};
+
+ Predicate predicate = new Predicate(">", attrAndValue);
+ Expression tsExpression = SparkV2Filters.convert(predicate);
+ Expression rawExpression = Expressions.greaterThan("x", epochMicros);
+
+ Assert.assertEquals(
+ "Generated Timestamp expression should be correct",
+ rawExpression.toString(),
+ tsExpression.toString());
+ }
+
+ @Test
+ public void testDateFilterConversion() {
+ LocalDate localDate = LocalDate.parse("2018-10-18");
+ long epochDay = localDate.toEpochDay();
+
+ NamedReference namedReference = FieldReference.apply("x");
+ LiteralValue ts = new LiteralValue(epochDay, DataTypes.DateType);
+ org.apache.spark.sql.connector.expressions.Expression[] attrAndValue =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference, ts};
+
+ Predicate predicate = new Predicate(">", attrAndValue);
+ Expression dateExpression = SparkV2Filters.convert(predicate);
+ Expression rawExpression = Expressions.greaterThan("x", epochDay);
+
+ Assert.assertEquals(
+ "Generated date expression should be correct",
+ rawExpression.toString(),
+ dateExpression.toString());
+ }
+
+ @Test
+ public void testNestedInInsideNot() {
+ NamedReference namedReference1 = FieldReference.apply("col1");
+ LiteralValue v1 = new LiteralValue(1, DataTypes.IntegerType);
+ LiteralValue v2 = new LiteralValue(2, DataTypes.IntegerType);
+ org.apache.spark.sql.connector.expressions.Expression[] attrAndValue1 =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference1, v1};
+ Predicate equal = new Predicate("=", attrAndValue1);
+
+ NamedReference namedReference2 = FieldReference.apply("col2");
+ org.apache.spark.sql.connector.expressions.Expression[] attrAndValue2 =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference2, v1, v2};
+ Predicate in = new Predicate("IN", attrAndValue2);
+
+ Not filter = new Not(new And(equal, in));
+ Expression converted = SparkV2Filters.convert(filter);
+ Assert.assertNull("Expression should not be converted", converted);
+ }
+
+ @Test
+ public void testNotIn() {
+ NamedReference namedReference = FieldReference.apply("col");
+ LiteralValue v1 = new LiteralValue(1, DataTypes.IntegerType);
+ LiteralValue v2 = new LiteralValue(2, DataTypes.IntegerType);
Review Comment:
Good catch! Will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r950644037
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/TestSparkV2Filters.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.temporal.ChronoUnit;
+import java.util.Map;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DataTypes;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestSparkV2Filters {
+
+ @Test
+ public void testV2Filters() {
+ Map<String, String> attrMap = Maps.newHashMap();
+ attrMap.put("id", "id");
+ attrMap.put("`i.d`", "i.d");
+ attrMap.put("`i``d`", "i`d");
+ attrMap.put("`d`.b.`dd```", "d.b.dd`");
+ attrMap.put("a.`aa```.c", "a.aa`.c");
+
+ attrMap.forEach(
+ (quoted, unquoted) -> {
+ NamedReference namedReference = FieldReference.apply(quoted);
+ org.apache.spark.sql.connector.expressions.Expression[] attrOnly =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference};
+
+ LiteralValue value = new LiteralValue(1, DataTypes.IntegerType);
+ org.apache.spark.sql.connector.expressions.Expression[] attrAndValue =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference, value};
+ org.apache.spark.sql.connector.expressions.Expression[] valueAndAttr =
+ new org.apache.spark.sql.connector.expressions.Expression[] {value, namedReference};
+
+ Predicate isNull = new Predicate("IS_NULL", attrOnly);
+ Expression expectedIsNull = Expressions.isNull(unquoted);
+ Expression actualIsNull = SparkV2Filters.convert(isNull);
+ Assert.assertEquals(
+ "IsNull must match", expectedIsNull.toString(), actualIsNull.toString());
+
+ Predicate isNotNull = new Predicate("IS_NOT_NULL", attrOnly);
+ Expression expectedIsNotNull = Expressions.notNull(unquoted);
+ Expression actualIsNotNull = SparkV2Filters.convert(isNotNull);
+ Assert.assertEquals(
+ "IsNotNull must match", expectedIsNotNull.toString(), actualIsNotNull.toString());
+
+ Predicate lt1 = new Predicate("<", attrAndValue);
+ Expression expectedLt1 = Expressions.lessThan(unquoted, 1);
+ Expression actualLt1 = SparkV2Filters.convert(lt1);
+ Assert.assertEquals("LessThan must match", expectedLt1.toString(), actualLt1.toString());
+
+ Predicate lt2 = new Predicate("<", valueAndAttr);
+ Expression expectedLt2 = Expressions.greaterThan(unquoted, 1);
+ Expression actualLt2 = SparkV2Filters.convert(lt2);
+ Assert.assertEquals("LessThan must match", expectedLt2.toString(), actualLt2.toString());
+
+ Predicate ltEq1 = new Predicate("<=", attrAndValue);
+ Expression expectedLtEq1 = Expressions.lessThanOrEqual(unquoted, 1);
+ Expression actualLtEq1 = SparkV2Filters.convert(ltEq1);
+ Assert.assertEquals(
+ "LessThanOrEqual must match", expectedLtEq1.toString(), actualLtEq1.toString());
+
+ Predicate ltEq2 = new Predicate("<=", valueAndAttr);
+ Expression expectedLtEq2 = Expressions.greaterThanOrEqual(unquoted, 1);
+ Expression actualLtEq2 = SparkV2Filters.convert(ltEq2);
+ Assert.assertEquals(
+ "LessThanOrEqual must match", expectedLtEq2.toString(), actualLtEq2.toString());
+
+ Predicate gt1 = new Predicate(">", attrAndValue);
+ Expression expectedGt1 = Expressions.greaterThan(unquoted, 1);
+ Expression actualGt1 = SparkV2Filters.convert(gt1);
+ Assert.assertEquals(
+ "GreaterThan must match", expectedGt1.toString(), actualGt1.toString());
+
+ Predicate gt2 = new Predicate(">", valueAndAttr);
+ Expression expectedGt2 = Expressions.lessThan(unquoted, 1);
+ Expression actualGt2 = SparkV2Filters.convert(gt2);
+ Assert.assertEquals(
+ "GreaterThan must match", expectedGt2.toString(), actualGt2.toString());
+
+ Predicate gtEq1 = new Predicate(">=", attrAndValue);
+ Expression expectedGtEq1 = Expressions.greaterThanOrEqual(unquoted, 1);
+ Expression actualGtEq1 = SparkV2Filters.convert(gtEq1);
+ Assert.assertEquals(
+ "GreaterThanOrEqual must match", expectedGtEq1.toString(), actualGtEq1.toString());
+
+ Predicate gtEq2 = new Predicate(">=", valueAndAttr);
+ Expression expectedGtEq2 = Expressions.lessThanOrEqual(unquoted, 1);
+ Expression actualGtEq2 = SparkV2Filters.convert(gtEq2);
+ Assert.assertEquals(
+ "GreaterThanOrEqual must match", expectedGtEq2.toString(), actualGtEq2.toString());
+
+ Predicate eq1 = new Predicate("=", attrAndValue);
+ Expression expectedEq1 = Expressions.equal(unquoted, 1);
+ Expression actualEq1 = SparkV2Filters.convert(eq1);
+ Assert.assertEquals("EqualTo must match", expectedEq1.toString(), actualEq1.toString());
+
+ Predicate eq2 = new Predicate("=", valueAndAttr);
+ Expression expectedEq2 = Expressions.equal(unquoted, 1);
+ Expression actualEq2 = SparkV2Filters.convert(eq2);
+ Assert.assertEquals("EqualTo must match", expectedEq2.toString(), actualEq2.toString());
+
+ Predicate eqNullSafe1 = new Predicate("<=>", attrAndValue);
+ Expression expectedEqNullSafe1 = Expressions.equal(unquoted, 1);
+ Expression actualEqNullSafe1 = SparkV2Filters.convert(eqNullSafe1);
+ Assert.assertEquals(
+ "EqualNullSafe must match",
+ expectedEqNullSafe1.toString(),
+ actualEqNullSafe1.toString());
+
+ Predicate eqNullSafe2 = new Predicate("<=>", valueAndAttr);
+ Expression expectedEqNullSafe2 = Expressions.equal(unquoted, 1);
+ Expression actualEqNullSafe2 = SparkV2Filters.convert(eqNullSafe2);
+ Assert.assertEquals(
+ "EqualNullSafe must match",
+ expectedEqNullSafe2.toString(),
+ actualEqNullSafe2.toString());
+
+ Predicate in = new Predicate("IN", attrAndValue);
+ Expression expectedIn = Expressions.in(unquoted, 1);
+ Expression actualIn = SparkV2Filters.convert(in);
+ Assert.assertEquals("In must match", expectedIn.toString(), actualIn.toString());
+
+ Predicate and = new And(lt1, eq1);
+ Expression expectedAnd = Expressions.and(expectedLt1, expectedEq1);
+ Expression actualAnd = SparkV2Filters.convert(and);
+ Assert.assertEquals("And must match", expectedAnd.toString(), actualAnd.toString());
+
+ Predicate invalid = new Predicate("<", attrOnly);
+ Predicate andWithInvalidLeft = new And(invalid, eq1);
+ Expression convertedAnd = SparkV2Filters.convert(andWithInvalidLeft);
+ Assert.assertEquals("And must match", convertedAnd, null);
+
+ Predicate or = new Or(lt1, eq1);
+ Expression expectedOr = Expressions.or(expectedLt1, expectedEq1);
+ Expression actualOr = SparkV2Filters.convert(or);
+ Assert.assertEquals("Or must match", expectedOr.toString(), actualOr.toString());
+
+ Predicate orWithInvalidLeft = new Or(invalid, eq1);
+ Expression convertedOr = SparkV2Filters.convert(orWithInvalidLeft);
+ Assert.assertEquals("Or must match", convertedOr, null);
+
+ Predicate not = new Not(lt1);
+ Expression expectedNot = Expressions.not(expectedLt1);
+ Expression actualNot = SparkV2Filters.convert(not);
+ Assert.assertEquals("Not must match", expectedNot.toString(), actualNot.toString());
+ });
+ }
+
+ @Test
+ public void testTimestampFilterConversion() {
+ Instant instant = Instant.parse("2018-10-18T00:00:57.907Z");
+ long epochMicros = ChronoUnit.MICROS.between(Instant.EPOCH, instant);
+
+ NamedReference namedReference = FieldReference.apply("x");
+ LiteralValue ts = new LiteralValue(epochMicros, DataTypes.TimestampType);
+ org.apache.spark.sql.connector.expressions.Expression[] attrAndValue =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference, ts};
+
+ Predicate predicate = new Predicate(">", attrAndValue);
+ Expression tsExpression = SparkV2Filters.convert(predicate);
+ Expression rawExpression = Expressions.greaterThan("x", epochMicros);
+
+ Assert.assertEquals(
+ "Generated Timestamp expression should be correct",
+ rawExpression.toString(),
+ tsExpression.toString());
+ }
+
+ @Test
+ public void testDateFilterConversion() {
+ LocalDate localDate = LocalDate.parse("2018-10-18");
+ long epochDay = localDate.toEpochDay();
+
+ NamedReference namedReference = FieldReference.apply("x");
+ LiteralValue ts = new LiteralValue(epochDay, DataTypes.DateType);
+ org.apache.spark.sql.connector.expressions.Expression[] attrAndValue =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference, ts};
+
+ Predicate predicate = new Predicate(">", attrAndValue);
+ Expression dateExpression = SparkV2Filters.convert(predicate);
+ Expression rawExpression = Expressions.greaterThan("x", epochDay);
+
+ Assert.assertEquals(
+ "Generated date expression should be correct",
+ rawExpression.toString(),
+ dateExpression.toString());
+ }
+
+ @Test
+ public void testNestedInInsideNot() {
+ NamedReference namedReference1 = FieldReference.apply("col1");
+ LiteralValue v1 = new LiteralValue(1, DataTypes.IntegerType);
+ LiteralValue v2 = new LiteralValue(2, DataTypes.IntegerType);
+ org.apache.spark.sql.connector.expressions.Expression[] attrAndValue1 =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference1, v1};
+ Predicate equal = new Predicate("=", attrAndValue1);
+
+ NamedReference namedReference2 = FieldReference.apply("col2");
+ org.apache.spark.sql.connector.expressions.Expression[] attrAndValue2 =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference2, v1, v2};
+ Predicate in = new Predicate("IN", attrAndValue2);
+
+ Not filter = new Not(new And(equal, in));
+ Expression converted = SparkV2Filters.convert(filter);
+ Assert.assertNull("Expression should not be converted", converted);
+ }
+
+ @Test
+ public void testNotIn() {
+ NamedReference namedReference = FieldReference.apply("col");
+ LiteralValue v1 = new LiteralValue(1, DataTypes.IntegerType);
+ LiteralValue v2 = new LiteralValue(2, DataTypes.IntegerType);
Review Comment:
Oh actually `Literal` is an abstract class, will have to use `LiteralValue` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
kbendick commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r950637743
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put(TRUE, Expression.Operation.TRUE)
+ .put(FALSE, Expression.Operation.FALSE)
+ .put(EQ, Expression.Operation.EQ)
+ .put(EQ_NULL_SAFE, Expression.Operation.EQ)
+ .put(GT, Expression.Operation.GT)
+ .put(GT_EQ, Expression.Operation.GT_EQ)
+ .put(LT, Expression.Operation.LT)
+ .put(LT_EQ, Expression.Operation.LT_EQ)
+ .put(IN, Expression.Operation.IN)
+ .put(IS_NULL, Expression.Operation.IS_NULL)
+ .put(NOT_NULL, Expression.Operation.NOT_NULL)
+ .put(AND, Expression.Operation.AND)
+ .put(OR, Expression.Operation.OR)
+ .put(NOT, Expression.Operation.NOT)
+ .put(STARTS_WITH, Expression.Operation.STARTS_WITH)
+ .build();
+
+ private static final int FIRST_ORDINAL = 0;
+ private static final int SECOND_ORDINAL = 1;
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
+
+ case LT:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[SECOND_ORDINAL]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ attributeName = predicate.children()[FIRST_ORDINAL].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = predicate.children()[SECOND_ORDINAL].toString();
+ value =
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value());
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
Review Comment:
Ah then absolutely keep it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#issuecomment-1228697212
Thank you all very much for your help!! @aokolnychyi @szehon-ho @kbendick
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r955556346
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isRef(child(predicate)) ? isNull(toColumnName(child(predicate))) : null;
+
+ case NOT_NULL:
+ return isRef(child(predicate)) ? notNull(toColumnName(child(predicate))) : null;
+
+ case LT:
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ String columnName = toColumnName(leftChild(predicate));
+ return lessThan(columnName, convertLiteral(rightChild(predicate)));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ String columnName = toColumnName(rightChild(predicate));
+ return greaterThan(columnName, convertLiteral(leftChild(predicate)));
+ } else {
+ return null;
+ }
+
+ case LT_EQ:
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ String columnName = toColumnName(leftChild(predicate));
+ return lessThanOrEqual(columnName, convertLiteral(rightChild(predicate)));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ String columnName = toColumnName(rightChild(predicate));
+ return greaterThanOrEqual(columnName, convertLiteral(leftChild(predicate)));
+ } else {
+ return null;
+ }
+
+ case GT:
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ String columnName = toColumnName(leftChild(predicate));
+ return greaterThan(columnName, convertLiteral(rightChild(predicate)));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ String columnName = toColumnName(rightChild(predicate));
+ return lessThan(columnName, convertLiteral(leftChild(predicate)));
+ } else {
+ return null;
+ }
+
+ case GT_EQ:
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ String columnName = toColumnName(leftChild(predicate));
+ return greaterThanOrEqual(columnName, convertLiteral(rightChild(predicate)));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ String columnName = toColumnName(rightChild(predicate));
+ return lessThanOrEqual(columnName, convertLiteral(leftChild(predicate)));
+ } else {
+ return null;
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String columnName;
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ columnName = toColumnName(leftChild(predicate));
+ value = convertLiteral(rightChild(predicate));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ columnName = toColumnName(rightChild(predicate));
+ value = convertLiteral(leftChild(predicate));
+ } else {
+ return null;
+ }
+
+ if (predicate.name().equals(EQ)) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(columnName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(columnName);
+ } else {
+ return handleEqual(columnName, value);
+ }
+ }
+
+ case IN:
+ if (isSupportedInPredicate(predicate)) {
+ return in(
+ toColumnName(childAtIndex(predicate, 0)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertLiteral(((Literal<?>) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ } else {
+ return null;
+ }
+
+ case NOT:
+ Not notPredicate = (Not) predicate;
+ Predicate childPredicate = notPredicate.child();
+ if (childPredicate.name().equals(IN)) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equal to notNull(col) && notIn(col, 1, 2) in Iceberg
+ Expression notIn =
+ notIn(
+ toColumnName(childAtIndex(childPredicate, 0)),
+ Arrays.stream(childPredicate.children())
+ .skip(1)
+ .map(val -> convertLiteral(((Literal<?>) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(toColumnName(childAtIndex(childPredicate, 0))), notIn);
+ } else if (hasNoInFilter(childPredicate)) {
+ Expression child = convert(childPredicate);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ if (isRef(leftChild(predicate))
Review Comment:
It's allowed by Spark but I think it's not realistic. It's not worthy to check. removed this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r947271363
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[0].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[0].toString()));
+
+ case LT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (predicate.children()[1] instanceof LiteralValue) {
+ attributeName = predicate.children()[0].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = predicate.children()[1].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value());
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(unquote(attributeName), value);
+ } else if (predicate.name().equals("<=>")) {
+ if (value == null) {
+ return isNull(unquote(attributeName));
+ } else {
+ return handleEqual(unquote(attributeName), value);
+ }
+ }
+ break;
Review Comment:
Nit: maybe we should return null explicitly like the other case instead of break. Also curious, will this ever happen?
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/TestSparkV2Filters.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.temporal.ChronoUnit;
+import java.util.Map;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DateType$;
+import org.apache.spark.sql.types.IntegerType$;
+import org.apache.spark.sql.types.TimestampType$;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestSparkV2Filters {
Review Comment:
Do we need some test of the cases of the expression the other way ( literal, attribute)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r936030720
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
Review Comment:
I think the main reason that Spark uses a String for the V2 Filter name is because Spark introduces [`GeneralScalarExpression`](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/GeneralScalarExpression.java) in 3.3 and pushes down a lot of functions such as `+`, `-`, `*`, `/`, `ABS`, `LOG`, ... Spark doesn't want to have a class for each of these Expressions so it uses a more genetic way. V2 [`Predicate`](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/filter/Predicate.java) is a subclass of `GeneralScalarExpression` so it also uses the field `name` for Filter name.
I am actually wondering if any of these function push down can benefit Iceberg. It seems to me that for both the data predicate push down and partition pruning, Iceberg only supports predicate `attri cmp lit`, so we can't really take advantage of any of these new function push down in Spark. In Spark built-in file source such as Parquet, partition pruning uses Catalyst Expression so Spark can prune partition such as `partCol1 + partCol2 > 3`.
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (checkIfPredicateValid(predicate) == null) {
+ return null;
Review Comment:
The idea is that if the Spark V2 filter is not valid, then we simply return a `null` and skip the convert code, then later on we don't need to bind and push down this predicate (https://github.com/apache/iceberg/blob/master/spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java#L120)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r954373220
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put(TRUE, Expression.Operation.TRUE)
+ .put(FALSE, Expression.Operation.FALSE)
+ .put(EQ, Expression.Operation.EQ)
+ .put(EQ_NULL_SAFE, Expression.Operation.EQ)
+ .put(GT, Expression.Operation.GT)
+ .put(GT_EQ, Expression.Operation.GT_EQ)
+ .put(LT, Expression.Operation.LT)
+ .put(LT_EQ, Expression.Operation.LT_EQ)
+ .put(IN, Expression.Operation.IN)
+ .put(IS_NULL, Expression.Operation.IS_NULL)
+ .put(NOT_NULL, Expression.Operation.NOT_NULL)
+ .put(AND, Expression.Operation.AND)
+ .put(OR, Expression.Operation.OR)
+ .put(NOT, Expression.Operation.NOT)
+ .put(STARTS_WITH, Expression.Operation.STARTS_WITH)
+ .build();
+
+ private static final int FIRST_ORDINAL = 0;
+ private static final int SECOND_ORDINAL = 1;
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
Review Comment:
Changed. Thanks for the suggestion!
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
Review Comment:
Added `childAtIndex`. Thanks for the suggestion!
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
Review Comment:
Fixed. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r950620473
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put(TRUE, Expression.Operation.TRUE)
+ .put(FALSE, Expression.Operation.FALSE)
+ .put(EQ, Expression.Operation.EQ)
+ .put(EQ_NULL_SAFE, Expression.Operation.EQ)
+ .put(GT, Expression.Operation.GT)
+ .put(GT_EQ, Expression.Operation.GT_EQ)
+ .put(LT, Expression.Operation.LT)
+ .put(LT_EQ, Expression.Operation.LT_EQ)
+ .put(IN, Expression.Operation.IN)
+ .put(IS_NULL, Expression.Operation.IS_NULL)
+ .put(NOT_NULL, Expression.Operation.NOT_NULL)
+ .put(AND, Expression.Operation.AND)
+ .put(OR, Expression.Operation.OR)
+ .put(NOT, Expression.Operation.NOT)
+ .put(STARTS_WITH, Expression.Operation.STARTS_WITH)
+ .build();
+
+ private static final int FIRST_ORDINAL = 0;
+ private static final int SECOND_ORDINAL = 1;
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
+
+ case LT:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[SECOND_ORDINAL]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ attributeName = predicate.children()[FIRST_ORDINAL].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = predicate.children()[SECOND_ORDINAL].toString();
+ value =
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value());
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(unquote(attributeName), value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(unquote(attributeName));
+ } else {
+ return handleEqual(unquote(attributeName), value);
+ }
+ }
+
+ case IN:
+ return in(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((LiteralValue) val).value()))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Expression notIn =
+ notIn(
+ unquote(childFilter.children()[FIRST_ORDINAL].toString()),
+ Arrays.stream(childFilter.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((LiteralValue) val).value()))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(unquote(childFilter.children()[FIRST_ORDINAL].toString())), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ {
+ return startsWith(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[SECOND_ORDINAL]).value())
+ .toString());
+ }
+ }
+ }
+
+ return null;
+ }
+
+ private static Object convertUTF8StringIfNecessary(Object value) {
Review Comment:
Can this method accept `Literal` from Spark and produce the Java object? We can then give a shorter and more generic name to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
kbendick commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r948144721
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[0].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[0].toString()));
+
+ case LT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
Review Comment:
Again, this might be overkill as they don't correspond much to anything meaningful (like the ones above in `TruncateFunction` do).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r948659578
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/TestSparkV2Filters.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.temporal.ChronoUnit;
+import java.util.Map;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DateType$;
+import org.apache.spark.sql.types.IntegerType$;
+import org.apache.spark.sql.types.TimestampType$;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestSparkV2Filters {
Review Comment:
I will refactor the tests in a follow up. Thanks for the suggestion!
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
Review Comment:
I have added named constants for the predicates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
kbendick commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r948137511
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/TestSparkV2Filters.java:
##########
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.temporal.ChronoUnit;
+import java.util.Map;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DateType$;
+import org.apache.spark.sql.types.IntegerType$;
+import org.apache.spark.sql.types.TimestampType$;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestSparkV2Filters {
+
+ @Test
+ public void testV2Filters() {
+ Map<String, String> attrMap = Maps.newHashMap();
+ attrMap.put("id", "id");
+ attrMap.put("`i.d`", "i.d");
+ attrMap.put("`i``d`", "i`d");
+ attrMap.put("`d`.b.`dd```", "d.b.dd`");
+ attrMap.put("a.`aa```.c", "a.aa`.c");
+
+ attrMap.forEach(
+ (quoted, unquoted) -> {
+ NamedReference namedReference = FieldReference.apply(quoted);
+ org.apache.spark.sql.connector.expressions.Expression[] attrOnly =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference};
+
+ LiteralValue value = new LiteralValue(1, IntegerType$.MODULE$);
+ org.apache.spark.sql.connector.expressions.Expression[] attrAndValue =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference, value};
+ org.apache.spark.sql.connector.expressions.Expression[] valueAndAttr =
+ new org.apache.spark.sql.connector.expressions.Expression[] {value, namedReference};
+
+ Predicate isNull = new Predicate("IS_NULL", attrOnly);
+ Expression expectedIsNull = Expressions.isNull(unquoted);
+ Expression actualIsNull = SparkV2Filters.convert(isNull);
+ Assert.assertEquals(
+ "IsNull must match", expectedIsNull.toString(), actualIsNull.toString());
+
+ Predicate isNotNull = new Predicate("IS_NOT_NULL", attrOnly);
+ Expression expectedIsNotNull = Expressions.notNull(unquoted);
+ Expression actualIsNotNull = SparkV2Filters.convert(isNotNull);
+ Assert.assertEquals(
+ "IsNotNull must match", expectedIsNotNull.toString(), actualIsNotNull.toString());
+
+ Predicate lt1 = new Predicate("<", attrAndValue);
+ Expression expectedLt1 = Expressions.lessThan(unquoted, 1);
+ Expression actualLt1 = SparkV2Filters.convert(lt1);
+ Assert.assertEquals("LessThan must match", expectedLt1.toString(), actualLt1.toString());
+
+ Predicate lt2 = new Predicate("<", valueAndAttr);
+ Expression expectedLt2 = Expressions.greaterThan(unquoted, 1);
+ Expression actualLt2 = SparkV2Filters.convert(lt2);
+ Assert.assertEquals("LessThan must match", expectedLt2.toString(), actualLt2.toString());
+
+ Predicate ltEq1 = new Predicate("<=", attrAndValue);
+ Expression expectedLtEq1 = Expressions.lessThanOrEqual(unquoted, 1);
+ Expression actualLtEq1 = SparkV2Filters.convert(ltEq1);
+ Assert.assertEquals(
+ "LessThanOrEqual must match", expectedLtEq1.toString(), actualLtEq1.toString());
+
+ Predicate ltEq2 = new Predicate("<=", valueAndAttr);
+ Expression expectedLtEq2 = Expressions.greaterThanOrEqual(unquoted, 1);
+ Expression actualLtEq2 = SparkV2Filters.convert(ltEq2);
+ Assert.assertEquals(
+ "LessThanOrEqual must match", expectedLtEq2.toString(), actualLtEq2.toString());
+
+ Predicate gt1 = new Predicate(">", attrAndValue);
+ Expression expectedGt1 = Expressions.greaterThan(unquoted, 1);
+ Expression actualGt1 = SparkV2Filters.convert(gt1);
+ Assert.assertEquals(
+ "GreaterThan must match", expectedGt1.toString(), actualGt1.toString());
+
+ Predicate gt2 = new Predicate(">", valueAndAttr);
+ Expression expectedGt2 = Expressions.lessThan(unquoted, 1);
+ Expression actualGt2 = SparkV2Filters.convert(gt2);
+ Assert.assertEquals(
+ "GreaterThan must match", expectedGt2.toString(), actualGt2.toString());
+
+ Predicate gtEq1 = new Predicate(">=", attrAndValue);
+ Expression expectedGtEq1 = Expressions.greaterThanOrEqual(unquoted, 1);
+ Expression actualGtEq1 = SparkV2Filters.convert(gtEq1);
+ Assert.assertEquals(
+ "GreaterThanOrEqual must match", expectedGtEq1.toString(), actualGtEq1.toString());
+
+ Predicate gtEq2 = new Predicate(">=", valueAndAttr);
+ Expression expectedGtEq2 = Expressions.lessThanOrEqual(unquoted, 1);
+ Expression actualGtEq2 = SparkV2Filters.convert(gtEq2);
+ Assert.assertEquals(
+ "GreaterThanOrEqual must match", expectedGtEq2.toString(), actualGtEq2.toString());
+
+ Predicate eq1 = new Predicate("=", attrAndValue);
+ Expression expectedEq1 = Expressions.equal(unquoted, 1);
+ Expression actualEq1 = SparkV2Filters.convert(eq1);
+ Assert.assertEquals("EqualTo must match", expectedEq1.toString(), actualEq1.toString());
+
+ Predicate eq2 = new Predicate("=", valueAndAttr);
+ Expression expectedEq2 = Expressions.equal(unquoted, 1);
+ Expression actualEq2 = SparkV2Filters.convert(eq2);
+ Assert.assertEquals("EqualTo must match", expectedEq2.toString(), actualEq2.toString());
+
+ Predicate eqNullSafe1 = new Predicate("<=>", attrAndValue);
+ Expression expectedEqNullSafe1 = Expressions.equal(unquoted, 1);
+ Expression actualEqNullSafe1 = SparkV2Filters.convert(eqNullSafe1);
+ Assert.assertEquals(
+ "EqualNullSafe must match",
+ expectedEqNullSafe1.toString(),
+ actualEqNullSafe1.toString());
+
+ Predicate eqNullSafe2 = new Predicate("<=>", valueAndAttr);
+ Expression expectedEqNullSafe2 = Expressions.equal(unquoted, 1);
+ Expression actualEqNullSafe2 = SparkV2Filters.convert(eqNullSafe2);
+ Assert.assertEquals(
+ "EqualNullSafe must match",
+ expectedEqNullSafe2.toString(),
+ actualEqNullSafe2.toString());
+
+ Predicate in = new Predicate("IN", attrAndValue);
+ Expression expectedIn = Expressions.in(unquoted, 1);
+ Expression actualIn = SparkV2Filters.convert(in);
+ Assert.assertEquals("In must match", expectedIn.toString(), actualIn.toString());
+
+ Predicate and = new And(lt1, eq1);
+ Expression expectedAnd = Expressions.and(expectedLt1, expectedEq1);
+ Expression actualAnd = SparkV2Filters.convert(and);
+ Assert.assertEquals("And must match", expectedAnd.toString(), actualAnd.toString());
+
+ Predicate invalid = new Predicate("<", attrOnly);
+ Predicate andWithInvalidLeft = new And(invalid, eq1);
+ Expression convertedAnd = SparkV2Filters.convert(andWithInvalidLeft);
+ Assert.assertEquals("And must match", convertedAnd, null);
+
+ Predicate or = new Or(lt1, eq1);
+ Expression expectedOr = Expressions.or(expectedLt1, expectedEq1);
+ Expression actualOr = SparkV2Filters.convert(or);
+ Assert.assertEquals("Or must match", expectedOr.toString(), actualOr.toString());
+
+ Predicate orWithInvalidLeft = new Or(invalid, eq1);
+ Expression convertedOr = SparkV2Filters.convert(orWithInvalidLeft);
+ Assert.assertEquals("Or must match", convertedOr, null);
+
+ Predicate not = new Not(lt1);
+ Expression expectedNot = Expressions.not(expectedLt1);
+ Expression actualNot = SparkV2Filters.convert(not);
+ Assert.assertEquals("Not must match", expectedNot.toString(), actualNot.toString());
+ });
+ }
+
+ @Test
+ public void testTimestampFilterConversion() {
+ Instant instant = Instant.parse("2018-10-18T00:00:57.907Z");
+ long epochMicros = ChronoUnit.MICROS.between(Instant.EPOCH, instant);
+
+ NamedReference namedReference = FieldReference.apply("x");
+ LiteralValue ts = new LiteralValue(epochMicros, TimestampType$.MODULE$);
+ org.apache.spark.sql.connector.expressions.Expression[] attrAndValue =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference, ts};
+
+ Predicate predicate = new Predicate(">", attrAndValue);
+ Expression tsExpression = SparkV2Filters.convert(predicate);
+ Expression rawExpression = Expressions.greaterThan("x", epochMicros);
+
+ Assert.assertEquals(
+ "Generated Timestamp expression should be correct",
+ rawExpression.toString(),
+ tsExpression.toString());
+ }
+
+ @Test
+ public void testDateFilterConversion() {
+ LocalDate localDate = LocalDate.parse("2018-10-18");
+ long epochDay = localDate.toEpochDay();
+
+ NamedReference namedReference = FieldReference.apply("x");
+ LiteralValue ts = new LiteralValue(epochDay, DateType$.MODULE$);
+ org.apache.spark.sql.connector.expressions.Expression[] attrAndValue =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference, ts};
+
+ Predicate predicate = new Predicate(">", attrAndValue);
+ Expression dateExpression = SparkV2Filters.convert(predicate);
+ Expression rawExpression = Expressions.greaterThan("x", epochDay);
+
+ Assert.assertEquals(
+ "Generated date expression should be correct",
+ rawExpression.toString(),
+ dateExpression.toString());
+ }
+
+ @Test
+ public void testNestedInInsideNot() {
+ NamedReference namedReference1 = FieldReference.apply("col1");
+ LiteralValue v1 = new LiteralValue(1, IntegerType$.MODULE$);
Review Comment:
Nit: I think `IntegerType$.Module$` is equivalent to `DataTypes.IntegerType`, is it not? They might be one dollar sign apart 🙃 but if they're equivalent we should use that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
kbendick commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r948133664
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[0].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[0].toString()));
+
+ case LT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (predicate.children()[1] instanceof LiteralValue) {
+ attributeName = predicate.children()[0].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = predicate.children()[1].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value());
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(unquote(attributeName), value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(unquote(attributeName));
+ } else {
+ return handleEqual(unquote(attributeName), value);
+ }
+ }
+
+ case IN:
+ return in(
+ unquote(predicate.children()[0].toString()),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((LiteralValue) val).value()))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Expression notIn =
+ notIn(
+ unquote(childFilter.children()[0].toString()),
+ Arrays.stream(childFilter.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((LiteralValue) val).value()))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(unquote(childFilter.children()[0].toString())), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ {
+ return startsWith(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value())
+ .toString());
+ }
+ }
+ }
+
+ return null;
+ }
+
+ private static Object convertUTF8StringIfNecessary(Object value) {
+ if (value instanceof UTF8String) {
+ return ((UTF8String) value).toString();
+ }
+ return value;
+ }
+
+ private static Expression handleEqual(String attribute, Object value) {
Review Comment:
Edit - Nevermind this is lifted from the original SparkFilters. Leaving it as is is the best choicee imo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r936140582
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (checkIfPredicateValid(predicate) == null) {
+ return null;
Review Comment:
I think this over, I probably should call `convert(Predicate predicate)` one by one in [SparkWriteBuilder.overwrite](https://github.com/apache/iceberg/blob/master/spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkWriteBuilder.java#L122). If I get null, then throw Exception saying `Cannot overwrite files using filters because not all the filter[s] can be converted correctly`. This is probably better than the error message `Cannot convert predicate to Iceberg`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r946952115
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (checkIfPredicateValid(predicate) == null) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[0].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[0].toString()));
+
+ case LT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
Review Comment:
We don't need this `convertLiteral` any more. This is the conversion that V2 filter is trying to avoid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
kbendick commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r948124519
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
Review Comment:
In a follow up, we should make named constants out of these to reduce the burden of using string types.
Even if they match our types, some of them potentially won't. And it would just be cleaner on the eyes imo.
Not a blocker or needed in this PR though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
kbendick commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r948126857
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
Review Comment:
Agreed. But I see why it's done.
The V2FunctionCatalog works the same way. Notice the `bucket` v2 function transform opt works the same way. With a string typed `numCols` and a string typed `bucket` function that Spark searches for on the implementing library's side. The name technically doesn't matter for the field but it 100% does for the function.
https://github.com/apache/spark/blob/126870eddc57fff7408942897dd59a2c19c91c4a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/V2ExpressionUtils.scala#L90-L100
I hope @huaxingao is right though that we will be able to use Spark to parse more expressions this way though =)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r948659834
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[0].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[0].toString()));
+
+ case LT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
Review Comment:
Fixed, but made the names a little shorter.
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[0].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[0].toString()));
+
+ case LT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (predicate.children()[1] instanceof LiteralValue) {
+ attributeName = predicate.children()[0].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = predicate.children()[1].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value());
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(unquote(attributeName), value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(unquote(attributeName));
+ } else {
+ return handleEqual(unquote(attributeName), value);
+ }
+ }
+
+ case IN:
+ return in(
+ unquote(predicate.children()[0].toString()),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((LiteralValue) val).value()))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Expression notIn =
+ notIn(
+ unquote(childFilter.children()[0].toString()),
+ Arrays.stream(childFilter.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((LiteralValue) val).value()))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(unquote(childFilter.children()[0].toString())), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ {
+ return startsWith(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value())
+ .toString());
+ }
+ }
+ }
+
+ return null;
+ }
+
+ private static Object convertUTF8StringIfNecessary(Object value) {
+ if (value instanceof UTF8String) {
+ return ((UTF8String) value).toString();
+ }
+ return value;
+ }
+
+ private static Expression handleEqual(String attribute, Object value) {
+ if (NaNUtil.isNaN(value)) {
+ return isNaN(attribute);
+ } else {
+ return equal(attribute, value);
+ }
+ }
+
+ private static String unquote(String attributeName) {
+ Matcher matcher = BACKTICKS_PATTERN.matcher(attributeName);
+ return matcher.replaceAll("$2");
+ }
+
+ private static boolean hasNoInFilter(Predicate predicate) {
+ Expression.Operation op = FILTERS.get(predicate.name());
+
+ if (op != null) {
+ switch (op) {
+ case AND:
+ And andPredicate = (And) predicate;
+ return hasNoInFilter(andPredicate.left()) && hasNoInFilter(andPredicate.right());
+ case OR:
+ Or orPredicate = (Or) predicate;
+ return hasNoInFilter(orPredicate.left()) && hasNoInFilter(orPredicate.right());
+ case NOT:
+ Not notPredicate = (Not) predicate;
+ return hasNoInFilter(notPredicate.child());
+ case IN:
+ return false;
+ default:
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ @SuppressWarnings("checkstyle:CyclomaticComplexity")
+ private static boolean valid(Predicate predicate) {
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case IS_NULL:
+ case NOT_NULL:
+ return predicate.children()[0] instanceof NamedReference;
+
+ case LT:
+ case LT_EQ:
+ case GT:
+ case GT_EQ:
+ case EQ:
+ if (predicate.children().length != 2) {
+ return false;
+ }
+ return (predicate.children()[0] instanceof NamedReference
+ && predicate.children()[1] instanceof LiteralValue)
Review Comment:
Fixed, but also made the names a little shorter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r950580132
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put(TRUE, Expression.Operation.TRUE)
+ .put(FALSE, Expression.Operation.FALSE)
+ .put(EQ, Expression.Operation.EQ)
+ .put(EQ_NULL_SAFE, Expression.Operation.EQ)
+ .put(GT, Expression.Operation.GT)
+ .put(GT_EQ, Expression.Operation.GT_EQ)
+ .put(LT, Expression.Operation.LT)
+ .put(LT_EQ, Expression.Operation.LT_EQ)
+ .put(IN, Expression.Operation.IN)
+ .put(IS_NULL, Expression.Operation.IS_NULL)
+ .put(NOT_NULL, Expression.Operation.NOT_NULL)
+ .put(AND, Expression.Operation.AND)
+ .put(OR, Expression.Operation.OR)
+ .put(NOT, Expression.Operation.NOT)
+ .put(STARTS_WITH, Expression.Operation.STARTS_WITH)
+ .build();
+
+ private static final int FIRST_ORDINAL = 0;
+ private static final int SECOND_ORDINAL = 1;
+
+ public static Expression convert(Predicate[] predicates) {
Review Comment:
What about removing this method for now? Our existing logic in `SparkFilters` is inconsistent right now as the similar method there does not do binding. We have to fix that first.
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
Review Comment:
nit: Can we do a direct import for `Operation` like in `SparkFilters`?
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put(TRUE, Expression.Operation.TRUE)
+ .put(FALSE, Expression.Operation.FALSE)
+ .put(EQ, Expression.Operation.EQ)
+ .put(EQ_NULL_SAFE, Expression.Operation.EQ)
+ .put(GT, Expression.Operation.GT)
+ .put(GT_EQ, Expression.Operation.GT_EQ)
+ .put(LT, Expression.Operation.LT)
+ .put(LT_EQ, Expression.Operation.LT_EQ)
+ .put(IN, Expression.Operation.IN)
+ .put(IS_NULL, Expression.Operation.IS_NULL)
+ .put(NOT_NULL, Expression.Operation.NOT_NULL)
+ .put(AND, Expression.Operation.AND)
+ .put(OR, Expression.Operation.OR)
+ .put(NOT, Expression.Operation.NOT)
+ .put(STARTS_WITH, Expression.Operation.STARTS_WITH)
+ .build();
+
+ private static final int FIRST_ORDINAL = 0;
+ private static final int SECOND_ORDINAL = 1;
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
+
+ case LT:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[SECOND_ORDINAL]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ attributeName = predicate.children()[FIRST_ORDINAL].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = predicate.children()[SECOND_ORDINAL].toString();
+ value =
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value());
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(unquote(attributeName), value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(unquote(attributeName));
+ } else {
+ return handleEqual(unquote(attributeName), value);
+ }
+ }
+
+ case IN:
+ return in(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((LiteralValue) val).value()))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Expression notIn =
+ notIn(
+ unquote(childFilter.children()[FIRST_ORDINAL].toString()),
+ Arrays.stream(childFilter.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((LiteralValue) val).value()))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(unquote(childFilter.children()[FIRST_ORDINAL].toString())), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
Review Comment:
nit: Do we need these extra `{}` here and 2 branches below?
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put(TRUE, Expression.Operation.TRUE)
+ .put(FALSE, Expression.Operation.FALSE)
+ .put(EQ, Expression.Operation.EQ)
+ .put(EQ_NULL_SAFE, Expression.Operation.EQ)
+ .put(GT, Expression.Operation.GT)
+ .put(GT_EQ, Expression.Operation.GT_EQ)
+ .put(LT, Expression.Operation.LT)
+ .put(LT_EQ, Expression.Operation.LT_EQ)
+ .put(IN, Expression.Operation.IN)
+ .put(IS_NULL, Expression.Operation.IS_NULL)
+ .put(NOT_NULL, Expression.Operation.NOT_NULL)
+ .put(AND, Expression.Operation.AND)
+ .put(OR, Expression.Operation.OR)
+ .put(NOT, Expression.Operation.NOT)
+ .put(STARTS_WITH, Expression.Operation.STARTS_WITH)
+ .build();
+
+ private static final int FIRST_ORDINAL = 0;
+ private static final int SECOND_ORDINAL = 1;
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
Review Comment:
@huaxingao, can you elaborate a bit on what is considered invalid in this case? Are we trying to validate the Spark logic or are we trying to check if the filter is supported by Iceberg?
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
Review Comment:
I think this follows what we did in `SparkFilters` but I agree with @kbendick that we should move this below all fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r950620473
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put(TRUE, Expression.Operation.TRUE)
+ .put(FALSE, Expression.Operation.FALSE)
+ .put(EQ, Expression.Operation.EQ)
+ .put(EQ_NULL_SAFE, Expression.Operation.EQ)
+ .put(GT, Expression.Operation.GT)
+ .put(GT_EQ, Expression.Operation.GT_EQ)
+ .put(LT, Expression.Operation.LT)
+ .put(LT_EQ, Expression.Operation.LT_EQ)
+ .put(IN, Expression.Operation.IN)
+ .put(IS_NULL, Expression.Operation.IS_NULL)
+ .put(NOT_NULL, Expression.Operation.NOT_NULL)
+ .put(AND, Expression.Operation.AND)
+ .put(OR, Expression.Operation.OR)
+ .put(NOT, Expression.Operation.NOT)
+ .put(STARTS_WITH, Expression.Operation.STARTS_WITH)
+ .build();
+
+ private static final int FIRST_ORDINAL = 0;
+ private static final int SECOND_ORDINAL = 1;
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
+
+ case LT:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[SECOND_ORDINAL]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ attributeName = predicate.children()[FIRST_ORDINAL].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = predicate.children()[SECOND_ORDINAL].toString();
+ value =
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value());
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(unquote(attributeName), value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(unquote(attributeName));
+ } else {
+ return handleEqual(unquote(attributeName), value);
+ }
+ }
+
+ case IN:
+ return in(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((LiteralValue) val).value()))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Expression notIn =
+ notIn(
+ unquote(childFilter.children()[FIRST_ORDINAL].toString()),
+ Arrays.stream(childFilter.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((LiteralValue) val).value()))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(unquote(childFilter.children()[FIRST_ORDINAL].toString())), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ {
+ return startsWith(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[SECOND_ORDINAL]).value())
+ .toString());
+ }
+ }
+ }
+
+ return null;
+ }
+
+ private static Object convertUTF8StringIfNecessary(Object value) {
Review Comment:
Can this method accept `Literal` from Spark and produce the Java object?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r923901658
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {
+ }
+
+ private static final Map<String, Expression.Operation> FILTERS = ImmutableMap
+ .<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (checkIfPredicateValid(predicate) == null) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(getAttrName(predicate.children()[0])));
+
+ case NOT_NULL:
+ return notNull(unquote(getAttrName(predicate.children()[0])));
+
+ case LT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThan(unquote(getAttrName(predicate.children()[0])),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(unquote(getAttrName(predicate.children()[1])),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
Review Comment:
I think this kind of normalization (e.g. `1 < a => a > 1`) should be done in Spark. I used to have this, but I just found out it was removed. I will add this back in Spark. But before this is available in Spark, I will have it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r935969257
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (checkIfPredicateValid(predicate) == null) {
+ return null;
Review Comment:
I'm a little confused why we use the return value here to signal error. The Code above iterates through the predicates one at a time so we don't collect all of the "non-convertable" predicates, I could see if we wanted to generate a large error message with all the filters we couldn't convert but it seems like we could just assert here rather than in the parent "convert" method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#issuecomment-1228689852
Thanks, @huaxingao! Thanks for reviewing, @szehon-ho @kbendick!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r954373726
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
Review Comment:
Fixed. Thanks!
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(attributeName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(attributeName);
+ } else {
+ return handleEqual(attributeName, value);
+ }
+ }
+
+ case IN:
+ return in(
+ toColumnName(leftChild(predicate)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Expression notIn =
+ notIn(
+ toColumnName(leftChild(childFilter)),
+ Arrays.stream(childFilter.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(toColumnName(leftChild(childFilter))), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ return startsWith(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)).toString());
+ }
+ }
+
+ return null;
+ }
+
+ private static Object convertUTF8StringIfNecessary(Literal literal) {
Review Comment:
Fixed. Thanks!
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(attributeName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(attributeName);
+ } else {
+ return handleEqual(attributeName, value);
+ }
+ }
+
+ case IN:
+ return in(
+ toColumnName(leftChild(predicate)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Expression notIn =
+ notIn(
+ toColumnName(leftChild(childFilter)),
+ Arrays.stream(childFilter.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(toColumnName(leftChild(childFilter))), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ return startsWith(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)).toString());
+ }
+ }
+
+ return null;
+ }
+
+ private static Object convertUTF8StringIfNecessary(Literal literal) {
+ if (literal.value() instanceof UTF8String) {
+ return ((UTF8String) literal.value()).toString();
+ }
+ return literal.value();
+ }
+
+ private static Expression handleEqual(String attribute, Object value) {
+ if (NaNUtil.isNaN(value)) {
+ return isNaN(attribute);
+ } else {
+ return equal(attribute, value);
+ }
+ }
+
+ private static boolean hasNoInFilter(Predicate predicate) {
+ Operation op = FILTERS.get(predicate.name());
+
+ if (op != null) {
+ switch (op) {
+ case AND:
+ And andPredicate = (And) predicate;
+ return hasNoInFilter(andPredicate.left()) && hasNoInFilter(andPredicate.right());
+ case OR:
+ Or orPredicate = (Or) predicate;
+ return hasNoInFilter(orPredicate.left()) && hasNoInFilter(orPredicate.right());
+ case NOT:
+ Not notPredicate = (Not) predicate;
+ return hasNoInFilter(notPredicate.child());
+ case IN:
+ return false;
+ default:
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ @SuppressWarnings("checkstyle:CyclomaticComplexity")
+ private static boolean valid(Predicate predicate) {
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case IS_NULL:
+ case NOT_NULL:
+ return child(predicate) instanceof NamedReference;
+
+ case LT:
+ case LT_EQ:
+ case GT:
+ case GT_EQ:
+ case EQ:
+ if (predicate.children().length != 2) {
+ return false;
+ }
+ return namedRefCompLit(predicate) || litCompNamedRef(predicate);
+
+ case IN:
+ if (!(leftChild(predicate) instanceof NamedReference)) {
+ return false;
+ } else {
+ return Arrays.stream(predicate.children())
+ .skip(1)
+ .allMatch(val -> val instanceof Literal);
+ }
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ return valid(notFilter.child());
+
+ case AND:
+ And andFilter = (And) predicate;
+ return valid(andFilter.left()) && valid(andFilter.right());
+
+ case OR:
+ Or orFilter = (Or) predicate;
+ return valid(orFilter.left()) && valid(orFilter.right());
+
+ case STARTS_WITH:
+ if (predicate.children().length != 2) {
+ return false;
+ }
+ return namedRefCompLit(predicate)
+ && ((Literal<?>) rightChild(predicate)).value() instanceof String;
Review Comment:
Good catch! It should be `UTF8String`
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/TestSparkV2Filters.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.temporal.ChronoUnit;
+import java.util.Map;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DataTypes;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestSparkV2Filters {
+
+ @Test
+ public void testV2Filters() {
Review Comment:
Added. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r955556151
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+
Review Comment:
Removed.
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isRef(child(predicate)) ? isNull(toColumnName(child(predicate))) : null;
+
+ case NOT_NULL:
+ return isRef(child(predicate)) ? notNull(toColumnName(child(predicate))) : null;
+
+ case LT:
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ String columnName = toColumnName(leftChild(predicate));
+ return lessThan(columnName, convertLiteral(rightChild(predicate)));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ String columnName = toColumnName(rightChild(predicate));
+ return greaterThan(columnName, convertLiteral(leftChild(predicate)));
+ } else {
+ return null;
+ }
+
+ case LT_EQ:
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ String columnName = toColumnName(leftChild(predicate));
+ return lessThanOrEqual(columnName, convertLiteral(rightChild(predicate)));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ String columnName = toColumnName(rightChild(predicate));
+ return greaterThanOrEqual(columnName, convertLiteral(leftChild(predicate)));
+ } else {
+ return null;
+ }
+
+ case GT:
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ String columnName = toColumnName(leftChild(predicate));
+ return greaterThan(columnName, convertLiteral(rightChild(predicate)));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ String columnName = toColumnName(rightChild(predicate));
+ return lessThan(columnName, convertLiteral(leftChild(predicate)));
+ } else {
+ return null;
+ }
+
+ case GT_EQ:
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ String columnName = toColumnName(leftChild(predicate));
+ return greaterThanOrEqual(columnName, convertLiteral(rightChild(predicate)));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ String columnName = toColumnName(rightChild(predicate));
+ return lessThanOrEqual(columnName, convertLiteral(leftChild(predicate)));
+ } else {
+ return null;
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String columnName;
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ columnName = toColumnName(leftChild(predicate));
+ value = convertLiteral(rightChild(predicate));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ columnName = toColumnName(rightChild(predicate));
+ value = convertLiteral(leftChild(predicate));
+ } else {
+ return null;
+ }
+
+ if (predicate.name().equals(EQ)) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(columnName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(columnName);
+ } else {
+ return handleEqual(columnName, value);
+ }
+ }
+
+ case IN:
+ if (isSupportedInPredicate(predicate)) {
+ return in(
+ toColumnName(childAtIndex(predicate, 0)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertLiteral(((Literal<?>) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ } else {
+ return null;
+ }
+
+ case NOT:
+ Not notPredicate = (Not) predicate;
+ Predicate childPredicate = notPredicate.child();
+ if (childPredicate.name().equals(IN)) {
Review Comment:
Added. Thanks!
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isRef(child(predicate)) ? isNull(toColumnName(child(predicate))) : null;
+
+ case NOT_NULL:
+ return isRef(child(predicate)) ? notNull(toColumnName(child(predicate))) : null;
+
+ case LT:
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ String columnName = toColumnName(leftChild(predicate));
+ return lessThan(columnName, convertLiteral(rightChild(predicate)));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ String columnName = toColumnName(rightChild(predicate));
+ return greaterThan(columnName, convertLiteral(leftChild(predicate)));
+ } else {
+ return null;
+ }
+
+ case LT_EQ:
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ String columnName = toColumnName(leftChild(predicate));
+ return lessThanOrEqual(columnName, convertLiteral(rightChild(predicate)));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ String columnName = toColumnName(rightChild(predicate));
+ return greaterThanOrEqual(columnName, convertLiteral(leftChild(predicate)));
+ } else {
+ return null;
+ }
+
+ case GT:
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ String columnName = toColumnName(leftChild(predicate));
+ return greaterThan(columnName, convertLiteral(rightChild(predicate)));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ String columnName = toColumnName(rightChild(predicate));
+ return lessThan(columnName, convertLiteral(leftChild(predicate)));
+ } else {
+ return null;
+ }
+
+ case GT_EQ:
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ String columnName = toColumnName(leftChild(predicate));
+ return greaterThanOrEqual(columnName, convertLiteral(rightChild(predicate)));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ String columnName = toColumnName(rightChild(predicate));
+ return lessThanOrEqual(columnName, convertLiteral(leftChild(predicate)));
+ } else {
+ return null;
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String columnName;
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ columnName = toColumnName(leftChild(predicate));
+ value = convertLiteral(rightChild(predicate));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ columnName = toColumnName(rightChild(predicate));
+ value = convertLiteral(leftChild(predicate));
+ } else {
+ return null;
+ }
+
+ if (predicate.name().equals(EQ)) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(columnName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(columnName);
+ } else {
+ return handleEqual(columnName, value);
+ }
+ }
+
+ case IN:
+ if (isSupportedInPredicate(predicate)) {
+ return in(
+ toColumnName(childAtIndex(predicate, 0)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertLiteral(((Literal<?>) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ } else {
+ return null;
+ }
+
+ case NOT:
+ Not notPredicate = (Not) predicate;
+ Predicate childPredicate = notPredicate.child();
+ if (childPredicate.name().equals(IN)) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equal to notNull(col) && notIn(col, 1, 2) in Iceberg
+ Expression notIn =
+ notIn(
+ toColumnName(childAtIndex(childPredicate, 0)),
+ Arrays.stream(childPredicate.children())
+ .skip(1)
+ .map(val -> convertLiteral(((Literal<?>) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(toColumnName(childAtIndex(childPredicate, 0))), notIn);
+ } else if (hasNoInFilter(childPredicate)) {
+ Expression child = convert(childPredicate);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ if (isRef(leftChild(predicate))
+ && isLiteral(rightChild(predicate))
+ && ((Literal<?>) rightChild(predicate)).value() instanceof UTF8String) {
+ String colName = toColumnName(leftChild(predicate));
+ return startsWith(colName, convertLiteral(rightChild(predicate)).toString());
+ } else {
+ return null;
+ }
+ }
+ }
+
+ return null;
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T childAtIndex(Predicate predicate, int index) {
+ return (T) predicate.children()[index];
+ }
+
+ private static boolean isRef(org.apache.spark.sql.connector.expressions.Expression expr) {
+ return expr instanceof NamedReference;
+ }
+
+ private static boolean isLiteral(org.apache.spark.sql.connector.expressions.Expression expr) {
+ return expr instanceof Literal;
+ }
+
+ private static Object convertLiteral(Literal<?> literal) {
+ if (literal.value() instanceof UTF8String) {
+ return ((UTF8String) literal.value()).toString();
+ }
+ return literal.value();
+ }
+
+ private static Expression handleEqual(String attribute, Object value) {
+ if (NaNUtil.isNaN(value)) {
+ return isNaN(attribute);
+ } else {
+ return equal(attribute, value);
+ }
+ }
+
+ private static boolean hasNoInFilter(Predicate predicate) {
+ Operation op = FILTERS.get(predicate.name());
+
+ if (op != null) {
+ switch (op) {
+ case AND:
+ And andPredicate = (And) predicate;
+ return hasNoInFilter(andPredicate.left()) && hasNoInFilter(andPredicate.right());
+ case OR:
+ Or orPredicate = (Or) predicate;
+ return hasNoInFilter(orPredicate.left()) && hasNoInFilter(orPredicate.right());
+ case NOT:
+ Not notPredicate = (Not) predicate;
+ return hasNoInFilter(notPredicate.child());
+ case IN:
+ return false;
+ default:
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ private static boolean isSupportedInPredicate(Predicate predicate) {
+ if (!isRef(childAtIndex(predicate, 0))) {
+ return false;
+ } else {
+ return Arrays.stream(predicate.children()).skip(1).allMatch(val -> isLiteral(val));
Review Comment:
Fixed. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
kbendick commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r948136237
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[0].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[0].toString()));
+
+ case LT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (predicate.children()[1] instanceof LiteralValue) {
+ attributeName = predicate.children()[0].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = predicate.children()[1].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value());
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(unquote(attributeName), value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(unquote(attributeName));
+ } else {
+ return handleEqual(unquote(attributeName), value);
+ }
+ }
+
+ case IN:
+ return in(
+ unquote(predicate.children()[0].toString()),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((LiteralValue) val).value()))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Expression notIn =
+ notIn(
+ unquote(childFilter.children()[0].toString()),
+ Arrays.stream(childFilter.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((LiteralValue) val).value()))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(unquote(childFilter.children()[0].toString())), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ {
+ return startsWith(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value())
+ .toString());
+ }
+ }
+ }
+
+ return null;
+ }
+
+ private static Object convertUTF8StringIfNecessary(Object value) {
+ if (value instanceof UTF8String) {
+ return ((UTF8String) value).toString();
+ }
+ return value;
+ }
+
+ private static Expression handleEqual(String attribute, Object value) {
+ if (NaNUtil.isNaN(value)) {
+ return isNaN(attribute);
+ } else {
+ return equal(attribute, value);
+ }
+ }
+
+ private static String unquote(String attributeName) {
+ Matcher matcher = BACKTICKS_PATTERN.matcher(attributeName);
+ return matcher.replaceAll("$2");
+ }
+
+ private static boolean hasNoInFilter(Predicate predicate) {
+ Expression.Operation op = FILTERS.get(predicate.name());
+
+ if (op != null) {
+ switch (op) {
+ case AND:
+ And andPredicate = (And) predicate;
+ return hasNoInFilter(andPredicate.left()) && hasNoInFilter(andPredicate.right());
+ case OR:
+ Or orPredicate = (Or) predicate;
+ return hasNoInFilter(orPredicate.left()) && hasNoInFilter(orPredicate.right());
+ case NOT:
+ Not notPredicate = (Not) predicate;
+ return hasNoInFilter(notPredicate.child());
+ case IN:
+ return false;
+ default:
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ @SuppressWarnings("checkstyle:CyclomaticComplexity")
+ private static boolean valid(Predicate predicate) {
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case IS_NULL:
+ case NOT_NULL:
+ return predicate.children()[0] instanceof NamedReference;
+
+ case LT:
+ case LT_EQ:
+ case GT:
+ case GT_EQ:
+ case EQ:
+ if (predicate.children().length != 2) {
+ return false;
+ }
+ return (predicate.children()[0] instanceof NamedReference
+ && predicate.children()[1] instanceof LiteralValue)
Review Comment:
This pattern comes up enough (and is somewhat confusing to read).
Maybe `binaryNamedReferenceComparedToLiteral` and `binaryLiterComparedToReference`? Though I guess if Spark is moving towards inlining with strings etc, it might be better to not try to deviate from that at first.
We can always refactor so I'd save something like this for a follow up PR (if at all). We'd probably wind up building up _a lot_ of functions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r950615128
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put(TRUE, Expression.Operation.TRUE)
+ .put(FALSE, Expression.Operation.FALSE)
+ .put(EQ, Expression.Operation.EQ)
+ .put(EQ_NULL_SAFE, Expression.Operation.EQ)
+ .put(GT, Expression.Operation.GT)
+ .put(GT_EQ, Expression.Operation.GT_EQ)
+ .put(LT, Expression.Operation.LT)
+ .put(LT_EQ, Expression.Operation.LT_EQ)
+ .put(IN, Expression.Operation.IN)
+ .put(IS_NULL, Expression.Operation.IS_NULL)
+ .put(NOT_NULL, Expression.Operation.NOT_NULL)
+ .put(AND, Expression.Operation.AND)
+ .put(OR, Expression.Operation.OR)
+ .put(NOT, Expression.Operation.NOT)
+ .put(STARTS_WITH, Expression.Operation.STARTS_WITH)
+ .build();
+
+ private static final int FIRST_ORDINAL = 0;
+ private static final int SECOND_ORDINAL = 1;
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
Review Comment:
Weird. I think my original comment wasn't posted.
I was saying that we shouldn't use unquoting workarounds we had for V1 filters. Instead, we should use field arrays in V2 predicates.
I'd consider something like this (needs to be polished further).
```
private static final Joiner DOT = Joiner.on(".");
...
@SuppressWarnings("unchecked")
private static <T> T child(Predicate predicate, int index) {
return (T) predicate.children()[index];
}
@SuppressWarnings("unchecked")
private static <T> T child(Predicate predicate) {
org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
Preconditions.checkArgument(children.length == 1, "TBD");
return (T) children[0];
}
private static String toColumnName(NamedReference ref) {
return DOT.join(ref.fieldNames());
}
```
And using it like this:
```
...
case IS_NULL:
return isNull(toColumnName(child(predicate)));
case NOT_NULL:
return notNull(toColumnName(child(predicate)));
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r950635566
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put(TRUE, Expression.Operation.TRUE)
+ .put(FALSE, Expression.Operation.FALSE)
+ .put(EQ, Expression.Operation.EQ)
+ .put(EQ_NULL_SAFE, Expression.Operation.EQ)
+ .put(GT, Expression.Operation.GT)
+ .put(GT_EQ, Expression.Operation.GT_EQ)
+ .put(LT, Expression.Operation.LT)
+ .put(LT_EQ, Expression.Operation.LT_EQ)
+ .put(IN, Expression.Operation.IN)
+ .put(IS_NULL, Expression.Operation.IS_NULL)
+ .put(NOT_NULL, Expression.Operation.NOT_NULL)
+ .put(AND, Expression.Operation.AND)
+ .put(OR, Expression.Operation.OR)
+ .put(NOT, Expression.Operation.NOT)
+ .put(STARTS_WITH, Expression.Operation.STARTS_WITH)
+ .build();
+
+ private static final int FIRST_ORDINAL = 0;
+ private static final int SECOND_ORDINAL = 1;
Review Comment:
I took @aokolnychyi's suggestion to use `leftChild` and `rightChild`
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
Review Comment:
Fixed. Thanks!
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put(TRUE, Expression.Operation.TRUE)
+ .put(FALSE, Expression.Operation.FALSE)
+ .put(EQ, Expression.Operation.EQ)
+ .put(EQ_NULL_SAFE, Expression.Operation.EQ)
+ .put(GT, Expression.Operation.GT)
+ .put(GT_EQ, Expression.Operation.GT_EQ)
+ .put(LT, Expression.Operation.LT)
+ .put(LT_EQ, Expression.Operation.LT_EQ)
+ .put(IN, Expression.Operation.IN)
+ .put(IS_NULL, Expression.Operation.IS_NULL)
+ .put(NOT_NULL, Expression.Operation.NOT_NULL)
+ .put(AND, Expression.Operation.AND)
+ .put(OR, Expression.Operation.OR)
+ .put(NOT, Expression.Operation.NOT)
+ .put(STARTS_WITH, Expression.Operation.STARTS_WITH)
+ .build();
+
+ private static final int FIRST_ORDINAL = 0;
+ private static final int SECOND_ORDINAL = 1;
+
+ public static Expression convert(Predicate[] predicates) {
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r955521117
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+
Review Comment:
nit: This empty line is probably unnecessary.
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isRef(child(predicate)) ? isNull(toColumnName(child(predicate))) : null;
+
+ case NOT_NULL:
+ return isRef(child(predicate)) ? notNull(toColumnName(child(predicate))) : null;
+
+ case LT:
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ String columnName = toColumnName(leftChild(predicate));
+ return lessThan(columnName, convertLiteral(rightChild(predicate)));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ String columnName = toColumnName(rightChild(predicate));
+ return greaterThan(columnName, convertLiteral(leftChild(predicate)));
+ } else {
+ return null;
+ }
+
+ case LT_EQ:
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ String columnName = toColumnName(leftChild(predicate));
+ return lessThanOrEqual(columnName, convertLiteral(rightChild(predicate)));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ String columnName = toColumnName(rightChild(predicate));
+ return greaterThanOrEqual(columnName, convertLiteral(leftChild(predicate)));
+ } else {
+ return null;
+ }
+
+ case GT:
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ String columnName = toColumnName(leftChild(predicate));
+ return greaterThan(columnName, convertLiteral(rightChild(predicate)));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ String columnName = toColumnName(rightChild(predicate));
+ return lessThan(columnName, convertLiteral(leftChild(predicate)));
+ } else {
+ return null;
+ }
+
+ case GT_EQ:
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ String columnName = toColumnName(leftChild(predicate));
+ return greaterThanOrEqual(columnName, convertLiteral(rightChild(predicate)));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ String columnName = toColumnName(rightChild(predicate));
+ return lessThanOrEqual(columnName, convertLiteral(leftChild(predicate)));
+ } else {
+ return null;
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String columnName;
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ columnName = toColumnName(leftChild(predicate));
+ value = convertLiteral(rightChild(predicate));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ columnName = toColumnName(rightChild(predicate));
+ value = convertLiteral(leftChild(predicate));
+ } else {
+ return null;
+ }
+
+ if (predicate.name().equals(EQ)) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(columnName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(columnName);
+ } else {
+ return handleEqual(columnName, value);
+ }
+ }
+
+ case IN:
+ if (isSupportedInPredicate(predicate)) {
+ return in(
+ toColumnName(childAtIndex(predicate, 0)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertLiteral(((Literal<?>) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ } else {
+ return null;
+ }
+
+ case NOT:
+ Not notPredicate = (Not) predicate;
+ Predicate childPredicate = notPredicate.child();
+ if (childPredicate.name().equals(IN)) {
Review Comment:
Shall we add also `isSupportedInPredicate(childPredicate)` to the condition?
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isRef(child(predicate)) ? isNull(toColumnName(child(predicate))) : null;
+
+ case NOT_NULL:
+ return isRef(child(predicate)) ? notNull(toColumnName(child(predicate))) : null;
+
+ case LT:
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ String columnName = toColumnName(leftChild(predicate));
+ return lessThan(columnName, convertLiteral(rightChild(predicate)));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ String columnName = toColumnName(rightChild(predicate));
+ return greaterThan(columnName, convertLiteral(leftChild(predicate)));
+ } else {
+ return null;
+ }
+
+ case LT_EQ:
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ String columnName = toColumnName(leftChild(predicate));
+ return lessThanOrEqual(columnName, convertLiteral(rightChild(predicate)));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ String columnName = toColumnName(rightChild(predicate));
+ return greaterThanOrEqual(columnName, convertLiteral(leftChild(predicate)));
+ } else {
+ return null;
+ }
+
+ case GT:
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ String columnName = toColumnName(leftChild(predicate));
+ return greaterThan(columnName, convertLiteral(rightChild(predicate)));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ String columnName = toColumnName(rightChild(predicate));
+ return lessThan(columnName, convertLiteral(leftChild(predicate)));
+ } else {
+ return null;
+ }
+
+ case GT_EQ:
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ String columnName = toColumnName(leftChild(predicate));
+ return greaterThanOrEqual(columnName, convertLiteral(rightChild(predicate)));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ String columnName = toColumnName(rightChild(predicate));
+ return lessThanOrEqual(columnName, convertLiteral(leftChild(predicate)));
+ } else {
+ return null;
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String columnName;
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ columnName = toColumnName(leftChild(predicate));
+ value = convertLiteral(rightChild(predicate));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ columnName = toColumnName(rightChild(predicate));
+ value = convertLiteral(leftChild(predicate));
+ } else {
+ return null;
+ }
+
+ if (predicate.name().equals(EQ)) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(columnName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(columnName);
+ } else {
+ return handleEqual(columnName, value);
+ }
+ }
+
+ case IN:
+ if (isSupportedInPredicate(predicate)) {
+ return in(
+ toColumnName(childAtIndex(predicate, 0)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertLiteral(((Literal<?>) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ } else {
+ return null;
+ }
+
+ case NOT:
+ Not notPredicate = (Not) predicate;
+ Predicate childPredicate = notPredicate.child();
+ if (childPredicate.name().equals(IN)) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equal to notNull(col) && notIn(col, 1, 2) in Iceberg
+ Expression notIn =
+ notIn(
+ toColumnName(childAtIndex(childPredicate, 0)),
+ Arrays.stream(childPredicate.children())
+ .skip(1)
+ .map(val -> convertLiteral(((Literal<?>) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(toColumnName(childAtIndex(childPredicate, 0))), notIn);
+ } else if (hasNoInFilter(childPredicate)) {
+ Expression child = convert(childPredicate);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ if (isRef(leftChild(predicate))
+ && isLiteral(rightChild(predicate))
+ && ((Literal<?>) rightChild(predicate)).value() instanceof UTF8String) {
+ String colName = toColumnName(leftChild(predicate));
+ return startsWith(colName, convertLiteral(rightChild(predicate)).toString());
+ } else {
+ return null;
+ }
+ }
+ }
+
+ return null;
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T childAtIndex(Predicate predicate, int index) {
+ return (T) predicate.children()[index];
+ }
+
+ private static boolean isRef(org.apache.spark.sql.connector.expressions.Expression expr) {
+ return expr instanceof NamedReference;
+ }
+
+ private static boolean isLiteral(org.apache.spark.sql.connector.expressions.Expression expr) {
+ return expr instanceof Literal;
+ }
+
+ private static Object convertLiteral(Literal<?> literal) {
+ if (literal.value() instanceof UTF8String) {
+ return ((UTF8String) literal.value()).toString();
+ }
+ return literal.value();
+ }
+
+ private static Expression handleEqual(String attribute, Object value) {
+ if (NaNUtil.isNaN(value)) {
+ return isNaN(attribute);
+ } else {
+ return equal(attribute, value);
+ }
+ }
+
+ private static boolean hasNoInFilter(Predicate predicate) {
+ Operation op = FILTERS.get(predicate.name());
+
+ if (op != null) {
+ switch (op) {
+ case AND:
+ And andPredicate = (And) predicate;
+ return hasNoInFilter(andPredicate.left()) && hasNoInFilter(andPredicate.right());
+ case OR:
+ Or orPredicate = (Or) predicate;
+ return hasNoInFilter(orPredicate.left()) && hasNoInFilter(orPredicate.right());
+ case NOT:
+ Not notPredicate = (Not) predicate;
+ return hasNoInFilter(notPredicate.child());
+ case IN:
+ return false;
+ default:
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ private static boolean isSupportedInPredicate(Predicate predicate) {
+ if (!isRef(childAtIndex(predicate, 0))) {
+ return false;
+ } else {
+ return Arrays.stream(predicate.children()).skip(1).allMatch(val -> isLiteral(val));
Review Comment:
nit: `val -> isLiteral(val)` -> `SparkV2Filters::isLiteral`?
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isRef(child(predicate)) ? isNull(toColumnName(child(predicate))) : null;
+
+ case NOT_NULL:
+ return isRef(child(predicate)) ? notNull(toColumnName(child(predicate))) : null;
+
+ case LT:
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ String columnName = toColumnName(leftChild(predicate));
+ return lessThan(columnName, convertLiteral(rightChild(predicate)));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ String columnName = toColumnName(rightChild(predicate));
+ return greaterThan(columnName, convertLiteral(leftChild(predicate)));
+ } else {
+ return null;
+ }
+
+ case LT_EQ:
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ String columnName = toColumnName(leftChild(predicate));
+ return lessThanOrEqual(columnName, convertLiteral(rightChild(predicate)));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ String columnName = toColumnName(rightChild(predicate));
+ return greaterThanOrEqual(columnName, convertLiteral(leftChild(predicate)));
+ } else {
+ return null;
+ }
+
+ case GT:
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ String columnName = toColumnName(leftChild(predicate));
+ return greaterThan(columnName, convertLiteral(rightChild(predicate)));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ String columnName = toColumnName(rightChild(predicate));
+ return lessThan(columnName, convertLiteral(leftChild(predicate)));
+ } else {
+ return null;
+ }
+
+ case GT_EQ:
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ String columnName = toColumnName(leftChild(predicate));
+ return greaterThanOrEqual(columnName, convertLiteral(rightChild(predicate)));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ String columnName = toColumnName(rightChild(predicate));
+ return lessThanOrEqual(columnName, convertLiteral(leftChild(predicate)));
+ } else {
+ return null;
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String columnName;
+ if (isRef(leftChild(predicate)) && isLiteral(rightChild(predicate))) {
+ columnName = toColumnName(leftChild(predicate));
+ value = convertLiteral(rightChild(predicate));
+ } else if (isRef(rightChild(predicate)) && isLiteral(leftChild(predicate))) {
+ columnName = toColumnName(rightChild(predicate));
+ value = convertLiteral(leftChild(predicate));
+ } else {
+ return null;
+ }
+
+ if (predicate.name().equals(EQ)) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(columnName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(columnName);
+ } else {
+ return handleEqual(columnName, value);
+ }
+ }
+
+ case IN:
+ if (isSupportedInPredicate(predicate)) {
+ return in(
+ toColumnName(childAtIndex(predicate, 0)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertLiteral(((Literal<?>) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ } else {
+ return null;
+ }
+
+ case NOT:
+ Not notPredicate = (Not) predicate;
+ Predicate childPredicate = notPredicate.child();
+ if (childPredicate.name().equals(IN)) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equal to notNull(col) && notIn(col, 1, 2) in Iceberg
+ Expression notIn =
+ notIn(
+ toColumnName(childAtIndex(childPredicate, 0)),
+ Arrays.stream(childPredicate.children())
+ .skip(1)
+ .map(val -> convertLiteral(((Literal<?>) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(toColumnName(childAtIndex(childPredicate, 0))), notIn);
+ } else if (hasNoInFilter(childPredicate)) {
+ Expression child = convert(childPredicate);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ if (isRef(leftChild(predicate))
Review Comment:
Is it realistic that Spark can pass something other than a reference followed by String? I think the other validation in this method is reasonable as we could technically have other valid expressions with structs inside IN or other predicates, which Iceberg does not support yet. I am not sure about this one, though.
I am fine keeping it too but I'd move it to `isSupportedStartsWithPredicate`. Up to you here, @huaxingao.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi merged pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
aokolnychyi merged PR #5302:
URL: https://github.com/apache/iceberg/pull/5302
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#issuecomment-1219078288
I'd like to take a look tomorrow or on Friday as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r950615128
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put(TRUE, Expression.Operation.TRUE)
+ .put(FALSE, Expression.Operation.FALSE)
+ .put(EQ, Expression.Operation.EQ)
+ .put(EQ_NULL_SAFE, Expression.Operation.EQ)
+ .put(GT, Expression.Operation.GT)
+ .put(GT_EQ, Expression.Operation.GT_EQ)
+ .put(LT, Expression.Operation.LT)
+ .put(LT_EQ, Expression.Operation.LT_EQ)
+ .put(IN, Expression.Operation.IN)
+ .put(IS_NULL, Expression.Operation.IS_NULL)
+ .put(NOT_NULL, Expression.Operation.NOT_NULL)
+ .put(AND, Expression.Operation.AND)
+ .put(OR, Expression.Operation.OR)
+ .put(NOT, Expression.Operation.NOT)
+ .put(STARTS_WITH, Expression.Operation.STARTS_WITH)
+ .build();
+
+ private static final int FIRST_ORDINAL = 0;
+ private static final int SECOND_ORDINAL = 1;
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
Review Comment:
Weird. I think my original comment wasn't posted.
I was saying that we shouldn't use unquoting workarounds we had for V1 filters. Instead, we should use field arrays in V2 predicates.
I'd consider something like this (needs to be polished further).
```
private static final Joiner DOT = Joiner.on(".");
...
@SuppressWarnings("unchecked")
private static <T> T child(Predicate predicate) {
org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
Preconditions.checkArgument(children.length == 1, "TBD");
return (T) children[0];
}
@SuppressWarnings("unchecked")
private static <T> T leftChild(Predicate predicate) {
return (T) ...
}
@SuppressWarnings("unchecked")
private static <T> T rightChild(Predicate predicate) {
return (T) ...
}
private static String toColumnName(NamedReference ref) {
return DOT.join(ref.fieldNames());
}
```
And using it like this:
```
...
case IS_NULL:
return isNull(toColumnName(child(predicate)));
case NOT_NULL:
return notNull(toColumnName(child(predicate)));
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r947335355
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[0].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[0].toString()));
+
+ case LT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (predicate.children()[1] instanceof LiteralValue) {
+ attributeName = predicate.children()[0].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = predicate.children()[1].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value());
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(unquote(attributeName), value);
+ } else if (predicate.name().equals("<=>")) {
+ if (value == null) {
+ return isNull(unquote(attributeName));
+ } else {
+ return handleEqual(unquote(attributeName), value);
+ }
+ }
+ break;
Review Comment:
This will not happen. I removed this.
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/TestSparkV2Filters.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.temporal.ChronoUnit;
+import java.util.Map;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DateType$;
+import org.apache.spark.sql.types.IntegerType$;
+import org.apache.spark.sql.types.TimestampType$;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestSparkV2Filters {
Review Comment:
Added some tests. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#issuecomment-1217285680
> any chance we can avoid having the normalization code in Iceberg, or will it take too long as we need to wait for a Spark pr and release?
I have already added the normalization code in Spark, but we need to wait for 3.4 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r935963144
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
Review Comment:
I find it kind of gross that Spark is making us deal with this, seems like this should be generated rather then us having to literally write out all the strings. Not that we have anything we can improve here. Still kind of hoping we can have some Substrait.io style solution for this kind of thing in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r923900082
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.spark;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {
+ }
+
+ private static final Map<String, Expression.Operation> FILTERS = ImmutableMap
+ .<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (checkIfPredicateValid(predicate) == null) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(getAttrName(predicate.children()[0])));
+
+ case NOT_NULL:
+ return notNull(unquote(getAttrName(predicate.children()[0])));
+
+ case LT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThan(unquote(getAttrName(predicate.children()[0])),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(unquote(getAttrName(predicate.children()[1])),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThanOrEqual(unquote(getAttrName(predicate.children()[0])),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(unquote(getAttrName(predicate.children()[1])),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThan(unquote(getAttrName(predicate.children()[0])),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThan(unquote(getAttrName(predicate.children()[1])),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThanOrEqual(unquote(getAttrName(predicate.children()[0])),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(unquote(getAttrName(predicate.children()[1])),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value = null;
+ String attributeName = "";
+ if (predicate.children()[1] instanceof LiteralValue) {
+ attributeName = getAttrName(predicate.children()[0]);
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = getAttrName(predicate.children()[1]);
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value());
+ }
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(value,
+ "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(unquote(attributeName), value);
+ } else if (predicate.name().equals("<=>")) {
+ if (value == null) {
+ return isNull(unquote(attributeName));
+ } else {
+ return handleEqual(unquote(attributeName), value);
+ }
+ }
+ break;
+
+ case IN:
+ Object[] inValues = new Object[predicate.children().length - 1];
+ for (int i = 1; i < predicate.children().length; i++) {
+ inValues[i - 1] = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[i]).value());
+ }
+ return in(unquote(getAttrName(predicate.children()[0])),
+ Stream.of(inValues)
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in Iceberg
+ Object[] notInValues = new Object[childFilter.children().length - 1];
+ for (int i = 1; i < childFilter.children().length; i++) {
+ notInValues[i - 1] = convertUTF8StringIfNecessary(((LiteralValue) childFilter.children()[i]).value());
+ }
+ Expression notIn = notIn(unquote(getAttrName(childFilter.children()[0])),
+ Stream.of(notInValues).collect(Collectors.toList()));
+ return and(notNull(unquote(getAttrName(childFilter.children()[0]))), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND: {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR: {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH: {
+ return startsWith(unquote(getAttrName(predicate.children()[0])),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()).toString());
+ }
+ }
+ }
+
+ return null;
+ }
+
+ private static Object convertUTF8StringIfNecessary(Object value) {
+ if (value instanceof UTF8String) {
+ return ((UTF8String) value).toString();
+ }
+ return value;
+ }
+
+ private static Expression handleEqual(String attribute, Object value) {
+ if (NaNUtil.isNaN(value)) {
+ return isNaN(attribute);
+ } else {
+ return equal(attribute, value);
+ }
+ }
+
+ private static String unquote(String attributeName) {
+ Matcher matcher = BACKTICKS_PATTERN.matcher(attributeName);
+ return matcher.replaceAll("$2");
+ }
+
+ private static boolean hasNoInFilter(Predicate predicate) {
+ Expression.Operation op = FILTERS.get(predicate.name());
+
+ if (op != null) {
+ switch (op) {
+ case AND:
+ And andPredicate = (And) predicate;
+ return hasNoInFilter(andPredicate.left()) && hasNoInFilter(andPredicate.right());
+ case OR:
+ Or orPredicate = (Or) predicate;
+ return hasNoInFilter(orPredicate.left()) && hasNoInFilter(orPredicate.right());
+ case NOT:
+ Not notPredicate = (Not) predicate;
+ return hasNoInFilter(notPredicate.child());
+ case IN:
+ return false;
+ default:
+ return true;
+ }
+ }
+
+ return false;
+ }
+
+ private static String getAttrName(org.apache.spark.sql.connector.expressions.Expression expr) {
+ return String.join(".", ((NamedReference) expr).fieldNames());
+ }
+
+ @SuppressWarnings("checkstyle:CyclomaticComplexity")
+ private static Predicate checkIfPredicateValid(Predicate predicate) {
Review Comment:
I think Spark should have this check to make sure all the V2 Filters are in valid format. Otherwise, this kind of invalid [filter](https://github.com/apache/iceberg/blob/master/spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java#L433) will get though.
I used to have this check on Spark side, but somehow it got removed during refactor. I will add this check back. Before this is available, I will have this check here. I will remove once this is in Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r947271363
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[0].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[0].toString()));
+
+ case LT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (predicate.children()[1] instanceof LiteralValue) {
+ attributeName = predicate.children()[0].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = predicate.children()[1].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value());
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(unquote(attributeName), value);
+ } else if (predicate.name().equals("<=>")) {
+ if (value == null) {
+ return isNull(unquote(attributeName));
+ } else {
+ return handleEqual(unquote(attributeName), value);
+ }
+ }
+ break;
Review Comment:
Nit: maybe we should return null explicitly like the other case instead of break, for consistency and to make explicit the unsupported case. Also curious, will this ever happen?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
kbendick commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r950639024
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/TestSparkV2Filters.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.temporal.ChronoUnit;
+import java.util.Map;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.spark.sql.connector.expressions.FieldReference;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.sql.types.DataTypes;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestSparkV2Filters {
+
+ @Test
+ public void testV2Filters() {
+ Map<String, String> attrMap = Maps.newHashMap();
+ attrMap.put("id", "id");
+ attrMap.put("`i.d`", "i.d");
+ attrMap.put("`i``d`", "i`d");
+ attrMap.put("`d`.b.`dd```", "d.b.dd`");
+ attrMap.put("a.`aa```.c", "a.aa`.c");
+
+ attrMap.forEach(
+ (quoted, unquoted) -> {
+ NamedReference namedReference = FieldReference.apply(quoted);
+ org.apache.spark.sql.connector.expressions.Expression[] attrOnly =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference};
+
+ LiteralValue value = new LiteralValue(1, DataTypes.IntegerType);
+ org.apache.spark.sql.connector.expressions.Expression[] attrAndValue =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference, value};
+ org.apache.spark.sql.connector.expressions.Expression[] valueAndAttr =
+ new org.apache.spark.sql.connector.expressions.Expression[] {value, namedReference};
+
+ Predicate isNull = new Predicate("IS_NULL", attrOnly);
+ Expression expectedIsNull = Expressions.isNull(unquoted);
+ Expression actualIsNull = SparkV2Filters.convert(isNull);
+ Assert.assertEquals(
+ "IsNull must match", expectedIsNull.toString(), actualIsNull.toString());
+
+ Predicate isNotNull = new Predicate("IS_NOT_NULL", attrOnly);
+ Expression expectedIsNotNull = Expressions.notNull(unquoted);
+ Expression actualIsNotNull = SparkV2Filters.convert(isNotNull);
+ Assert.assertEquals(
+ "IsNotNull must match", expectedIsNotNull.toString(), actualIsNotNull.toString());
+
+ Predicate lt1 = new Predicate("<", attrAndValue);
+ Expression expectedLt1 = Expressions.lessThan(unquoted, 1);
+ Expression actualLt1 = SparkV2Filters.convert(lt1);
+ Assert.assertEquals("LessThan must match", expectedLt1.toString(), actualLt1.toString());
+
+ Predicate lt2 = new Predicate("<", valueAndAttr);
+ Expression expectedLt2 = Expressions.greaterThan(unquoted, 1);
+ Expression actualLt2 = SparkV2Filters.convert(lt2);
+ Assert.assertEquals("LessThan must match", expectedLt2.toString(), actualLt2.toString());
+
+ Predicate ltEq1 = new Predicate("<=", attrAndValue);
+ Expression expectedLtEq1 = Expressions.lessThanOrEqual(unquoted, 1);
+ Expression actualLtEq1 = SparkV2Filters.convert(ltEq1);
+ Assert.assertEquals(
+ "LessThanOrEqual must match", expectedLtEq1.toString(), actualLtEq1.toString());
+
+ Predicate ltEq2 = new Predicate("<=", valueAndAttr);
+ Expression expectedLtEq2 = Expressions.greaterThanOrEqual(unquoted, 1);
+ Expression actualLtEq2 = SparkV2Filters.convert(ltEq2);
+ Assert.assertEquals(
+ "LessThanOrEqual must match", expectedLtEq2.toString(), actualLtEq2.toString());
+
+ Predicate gt1 = new Predicate(">", attrAndValue);
+ Expression expectedGt1 = Expressions.greaterThan(unquoted, 1);
+ Expression actualGt1 = SparkV2Filters.convert(gt1);
+ Assert.assertEquals(
+ "GreaterThan must match", expectedGt1.toString(), actualGt1.toString());
+
+ Predicate gt2 = new Predicate(">", valueAndAttr);
+ Expression expectedGt2 = Expressions.lessThan(unquoted, 1);
+ Expression actualGt2 = SparkV2Filters.convert(gt2);
+ Assert.assertEquals(
+ "GreaterThan must match", expectedGt2.toString(), actualGt2.toString());
+
+ Predicate gtEq1 = new Predicate(">=", attrAndValue);
+ Expression expectedGtEq1 = Expressions.greaterThanOrEqual(unquoted, 1);
+ Expression actualGtEq1 = SparkV2Filters.convert(gtEq1);
+ Assert.assertEquals(
+ "GreaterThanOrEqual must match", expectedGtEq1.toString(), actualGtEq1.toString());
+
+ Predicate gtEq2 = new Predicate(">=", valueAndAttr);
+ Expression expectedGtEq2 = Expressions.lessThanOrEqual(unquoted, 1);
+ Expression actualGtEq2 = SparkV2Filters.convert(gtEq2);
+ Assert.assertEquals(
+ "GreaterThanOrEqual must match", expectedGtEq2.toString(), actualGtEq2.toString());
+
+ Predicate eq1 = new Predicate("=", attrAndValue);
+ Expression expectedEq1 = Expressions.equal(unquoted, 1);
+ Expression actualEq1 = SparkV2Filters.convert(eq1);
+ Assert.assertEquals("EqualTo must match", expectedEq1.toString(), actualEq1.toString());
+
+ Predicate eq2 = new Predicate("=", valueAndAttr);
+ Expression expectedEq2 = Expressions.equal(unquoted, 1);
+ Expression actualEq2 = SparkV2Filters.convert(eq2);
+ Assert.assertEquals("EqualTo must match", expectedEq2.toString(), actualEq2.toString());
+
+ Predicate eqNullSafe1 = new Predicate("<=>", attrAndValue);
+ Expression expectedEqNullSafe1 = Expressions.equal(unquoted, 1);
+ Expression actualEqNullSafe1 = SparkV2Filters.convert(eqNullSafe1);
+ Assert.assertEquals(
+ "EqualNullSafe must match",
+ expectedEqNullSafe1.toString(),
+ actualEqNullSafe1.toString());
+
+ Predicate eqNullSafe2 = new Predicate("<=>", valueAndAttr);
+ Expression expectedEqNullSafe2 = Expressions.equal(unquoted, 1);
+ Expression actualEqNullSafe2 = SparkV2Filters.convert(eqNullSafe2);
+ Assert.assertEquals(
+ "EqualNullSafe must match",
+ expectedEqNullSafe2.toString(),
+ actualEqNullSafe2.toString());
+
+ Predicate in = new Predicate("IN", attrAndValue);
+ Expression expectedIn = Expressions.in(unquoted, 1);
+ Expression actualIn = SparkV2Filters.convert(in);
+ Assert.assertEquals("In must match", expectedIn.toString(), actualIn.toString());
+
+ Predicate and = new And(lt1, eq1);
+ Expression expectedAnd = Expressions.and(expectedLt1, expectedEq1);
+ Expression actualAnd = SparkV2Filters.convert(and);
+ Assert.assertEquals("And must match", expectedAnd.toString(), actualAnd.toString());
+
+ Predicate invalid = new Predicate("<", attrOnly);
+ Predicate andWithInvalidLeft = new And(invalid, eq1);
+ Expression convertedAnd = SparkV2Filters.convert(andWithInvalidLeft);
+ Assert.assertEquals("And must match", convertedAnd, null);
+
+ Predicate or = new Or(lt1, eq1);
+ Expression expectedOr = Expressions.or(expectedLt1, expectedEq1);
+ Expression actualOr = SparkV2Filters.convert(or);
+ Assert.assertEquals("Or must match", expectedOr.toString(), actualOr.toString());
+
+ Predicate orWithInvalidLeft = new Or(invalid, eq1);
+ Expression convertedOr = SparkV2Filters.convert(orWithInvalidLeft);
+ Assert.assertEquals("Or must match", convertedOr, null);
+
+ Predicate not = new Not(lt1);
+ Expression expectedNot = Expressions.not(expectedLt1);
+ Expression actualNot = SparkV2Filters.convert(not);
+ Assert.assertEquals("Not must match", expectedNot.toString(), actualNot.toString());
+ });
+ }
+
+ @Test
+ public void testTimestampFilterConversion() {
+ Instant instant = Instant.parse("2018-10-18T00:00:57.907Z");
+ long epochMicros = ChronoUnit.MICROS.between(Instant.EPOCH, instant);
+
+ NamedReference namedReference = FieldReference.apply("x");
+ LiteralValue ts = new LiteralValue(epochMicros, DataTypes.TimestampType);
+ org.apache.spark.sql.connector.expressions.Expression[] attrAndValue =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference, ts};
+
+ Predicate predicate = new Predicate(">", attrAndValue);
+ Expression tsExpression = SparkV2Filters.convert(predicate);
+ Expression rawExpression = Expressions.greaterThan("x", epochMicros);
+
+ Assert.assertEquals(
+ "Generated Timestamp expression should be correct",
+ rawExpression.toString(),
+ tsExpression.toString());
+ }
+
+ @Test
+ public void testDateFilterConversion() {
+ LocalDate localDate = LocalDate.parse("2018-10-18");
+ long epochDay = localDate.toEpochDay();
+
+ NamedReference namedReference = FieldReference.apply("x");
+ LiteralValue ts = new LiteralValue(epochDay, DataTypes.DateType);
+ org.apache.spark.sql.connector.expressions.Expression[] attrAndValue =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference, ts};
+
+ Predicate predicate = new Predicate(">", attrAndValue);
+ Expression dateExpression = SparkV2Filters.convert(predicate);
+ Expression rawExpression = Expressions.greaterThan("x", epochDay);
+
+ Assert.assertEquals(
+ "Generated date expression should be correct",
+ rawExpression.toString(),
+ dateExpression.toString());
+ }
+
+ @Test
+ public void testNestedInInsideNot() {
+ NamedReference namedReference1 = FieldReference.apply("col1");
+ LiteralValue v1 = new LiteralValue(1, DataTypes.IntegerType);
+ LiteralValue v2 = new LiteralValue(2, DataTypes.IntegerType);
+ org.apache.spark.sql.connector.expressions.Expression[] attrAndValue1 =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference1, v1};
+ Predicate equal = new Predicate("=", attrAndValue1);
+
+ NamedReference namedReference2 = FieldReference.apply("col2");
+ org.apache.spark.sql.connector.expressions.Expression[] attrAndValue2 =
+ new org.apache.spark.sql.connector.expressions.Expression[] {namedReference2, v1, v2};
+ Predicate in = new Predicate("IN", attrAndValue2);
+
+ Not filter = new Not(new And(equal, in));
+ Expression converted = SparkV2Filters.convert(filter);
+ Assert.assertNull("Expression should not be converted", converted);
+ }
+
+ @Test
+ public void testNotIn() {
+ NamedReference namedReference = FieldReference.apply("col");
+ LiteralValue v1 = new LiteralValue(1, DataTypes.IntegerType);
+ LiteralValue v2 = new LiteralValue(2, DataTypes.IntegerType);
Review Comment:
Nit: Can `Literal` be used in place of `LiteralValue` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r950635547
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
Review Comment:
Fixed. Thanks!
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put(TRUE, Expression.Operation.TRUE)
+ .put(FALSE, Expression.Operation.FALSE)
+ .put(EQ, Expression.Operation.EQ)
+ .put(EQ_NULL_SAFE, Expression.Operation.EQ)
+ .put(GT, Expression.Operation.GT)
+ .put(GT_EQ, Expression.Operation.GT_EQ)
+ .put(LT, Expression.Operation.LT)
+ .put(LT_EQ, Expression.Operation.LT_EQ)
+ .put(IN, Expression.Operation.IN)
+ .put(IS_NULL, Expression.Operation.IS_NULL)
+ .put(NOT_NULL, Expression.Operation.NOT_NULL)
+ .put(AND, Expression.Operation.AND)
+ .put(OR, Expression.Operation.OR)
+ .put(NOT, Expression.Operation.NOT)
+ .put(STARTS_WITH, Expression.Operation.STARTS_WITH)
+ .build();
+
+ private static final int FIRST_ORDINAL = 0;
+ private static final int SECOND_ORDINAL = 1;
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[FIRST_ORDINAL].toString()));
+
+ case LT:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[SECOND_ORDINAL]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[FIRST_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[SECOND_ORDINAL].toString()),
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (predicate.children()[SECOND_ORDINAL] instanceof LiteralValue) {
+ attributeName = predicate.children()[FIRST_ORDINAL].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = predicate.children()[SECOND_ORDINAL].toString();
+ value =
+ convertUTF8StringIfNecessary(
+ ((LiteralValue) predicate.children()[FIRST_ORDINAL]).value());
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
Review Comment:
This is actually borrowed from `SparkFilters`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r954373366
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
Review Comment:
Fixed. Thanks for the suggestion!
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
Review Comment:
Fixed. Thanks for the suggestion!
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expression.Operation;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Joiner;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.Literal;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Joiner DOT = Joiner.on(".");
+
+ private static final String TRUE = "ALWAYS_TRUE";
+ private static final String FALSE = "ALWAYS_FALSE";
+ private static final String EQ = "=";
+ private static final String EQ_NULL_SAFE = "<=>";
+ private static final String GT = ">";
+ private static final String GT_EQ = ">=";
+ private static final String LT = "<";
+ private static final String LT_EQ = "<=";
+ private static final String IN = "IN";
+ private static final String IS_NULL = "IS_NULL";
+ private static final String NOT_NULL = "IS_NOT_NULL";
+ private static final String AND = "AND";
+ private static final String OR = "OR";
+ private static final String NOT = "NOT";
+ private static final String STARTS_WITH = "STARTS_WITH";
+
+ private static final Map<String, Operation> FILTERS =
+ ImmutableMap.<String, Operation>builder()
+ .put(TRUE, Operation.TRUE)
+ .put(FALSE, Operation.FALSE)
+ .put(EQ, Operation.EQ)
+ .put(EQ_NULL_SAFE, Operation.EQ)
+ .put(GT, Operation.GT)
+ .put(GT_EQ, Operation.GT_EQ)
+ .put(LT, Operation.LT)
+ .put(LT_EQ, Operation.LT_EQ)
+ .put(IN, Operation.IN)
+ .put(IS_NULL, Operation.IS_NULL)
+ .put(NOT_NULL, Operation.NOT_NULL)
+ .put(AND, Operation.AND)
+ .put(OR, Operation.OR)
+ .put(NOT, Operation.NOT)
+ .put(STARTS_WITH, Operation.STARTS_WITH)
+ .build();
+
+ private SparkV2Filters() {}
+
+ @SuppressWarnings("unchecked")
+ private static <T> T child(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 1, "Predicate should have one child: %s", predicate);
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T leftChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ return (T) children[0];
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T> T rightChild(Predicate predicate) {
+ org.apache.spark.sql.connector.expressions.Expression[] children = predicate.children();
+ Preconditions.checkArgument(
+ children.length == 2, "Predicate should have two children: %s", predicate);
+ return (T) children[1];
+ }
+
+ private static String toColumnName(NamedReference ref) {
+ return DOT.join(ref.fieldNames());
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(toColumnName(child(predicate)));
+
+ case NOT_NULL:
+ return notNull(toColumnName(child(predicate)));
+
+ case LT:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case LT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return lessThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return greaterThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThan(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThan(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case GT_EQ:
+ if (rightChild(predicate) instanceof Literal) {
+ return greaterThanOrEqual(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)));
+ } else {
+ return lessThanOrEqual(
+ toColumnName(rightChild(predicate)),
+ convertUTF8StringIfNecessary(leftChild(predicate)));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (rightChild(predicate) instanceof Literal) {
+ attributeName = toColumnName(leftChild(predicate));
+ value = convertUTF8StringIfNecessary(rightChild(predicate));
+ } else {
+ attributeName = toColumnName(rightChild(predicate));
+ value = convertUTF8StringIfNecessary(leftChild(predicate));
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(attributeName, value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(attributeName);
+ } else {
+ return handleEqual(attributeName, value);
+ }
+ }
+
+ case IN:
+ return in(
+ toColumnName(leftChild(predicate)),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Expression notIn =
+ notIn(
+ toColumnName(leftChild(childFilter)),
+ Arrays.stream(childFilter.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((Literal) val)))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(toColumnName(leftChild(childFilter))), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ return startsWith(
+ toColumnName(leftChild(predicate)),
+ convertUTF8StringIfNecessary(rightChild(predicate)).toString());
+ }
+ }
+
+ return null;
+ }
+
+ private static Object convertUTF8StringIfNecessary(Literal literal) {
Review Comment:
Changed. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
kbendick commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r948131736
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (!valid(predicate)) {
+ return null;
+ }
+
+ Expression.Operation op = FILTERS.get(predicate.name());
+ if (op != null) {
+ switch (op) {
+ case TRUE:
+ return Expressions.alwaysTrue();
+
+ case FALSE:
+ return Expressions.alwaysFalse();
+
+ case IS_NULL:
+ return isNull(unquote(predicate.children()[0].toString()));
+
+ case NOT_NULL:
+ return notNull(unquote(predicate.children()[0].toString()));
+
+ case LT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case LT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return lessThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThan(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThan(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case GT_EQ:
+ if (predicate.children()[1] instanceof LiteralValue) {
+ return greaterThanOrEqual(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value()));
+ } else {
+ return lessThanOrEqual(
+ unquote(predicate.children()[1].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value()));
+ }
+
+ case EQ: // used for both eq and null-safe-eq
+ Object value;
+ String attributeName;
+ if (predicate.children()[1] instanceof LiteralValue) {
+ attributeName = predicate.children()[0].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value());
+ } else {
+ attributeName = predicate.children()[1].toString();
+ value = convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[0]).value());
+ }
+
+ if (predicate.name().equals("=")) {
+ // comparison with null in normal equality is always null. this is probably a mistake.
+ Preconditions.checkNotNull(
+ value, "Expression is always false (eq is not null-safe): %s", predicate);
+ return handleEqual(unquote(attributeName), value);
+ } else { // "<=>"
+ if (value == null) {
+ return isNull(unquote(attributeName));
+ } else {
+ return handleEqual(unquote(attributeName), value);
+ }
+ }
+
+ case IN:
+ return in(
+ unquote(predicate.children()[0].toString()),
+ Arrays.stream(predicate.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((LiteralValue) val).value()))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+
+ case NOT:
+ Not notFilter = (Not) predicate;
+ Predicate childFilter = notFilter.child();
+ if (childFilter.name().equals("IN")) {
+ // infer an extra notNull predicate for Spark NOT IN filters
+ // as Iceberg expressions don't follow the 3-value SQL boolean logic
+ // col NOT IN (1, 2) in Spark is equivalent to notNull(col) && notIn(col, 1, 2) in
+ // Iceberg
+ Expression notIn =
+ notIn(
+ unquote(childFilter.children()[0].toString()),
+ Arrays.stream(childFilter.children())
+ .skip(1)
+ .map(val -> convertUTF8StringIfNecessary(((LiteralValue) val).value()))
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList()));
+ return and(notNull(unquote(childFilter.children()[0].toString())), notIn);
+ } else if (hasNoInFilter(childFilter)) {
+ Expression child = convert(childFilter);
+ if (child != null) {
+ return not(child);
+ }
+ }
+ return null;
+
+ case AND:
+ {
+ And andPredicate = (And) predicate;
+ Expression left = convert(andPredicate.left());
+ Expression right = convert(andPredicate.right());
+ if (left != null && right != null) {
+ return and(left, right);
+ }
+ return null;
+ }
+
+ case OR:
+ {
+ Or orPredicate = (Or) predicate;
+ Expression left = convert(orPredicate.left());
+ Expression right = convert(orPredicate.right());
+ if (left != null && right != null) {
+ return or(left, right);
+ }
+ return null;
+ }
+
+ case STARTS_WITH:
+ {
+ return startsWith(
+ unquote(predicate.children()[0].toString()),
+ convertUTF8StringIfNecessary(((LiteralValue) predicate.children()[1]).value())
+ .toString());
+ }
+ }
+ }
+
+ return null;
+ }
+
+ private static Object convertUTF8StringIfNecessary(Object value) {
+ if (value instanceof UTF8String) {
+ return ((UTF8String) value).toString();
+ }
+ return value;
+ }
+
+ private static Expression handleEqual(String attribute, Object value) {
Review Comment:
Maybe just `NaNSafeEquals` or `SafeEquals`. This is fine as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
kbendick commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r948126857
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
Review Comment:
Agreed. But I see why it's done.
The V2FunctionCatalog works the same way. Notice the `bucket` v2 function transform opt works the same way. With a string typed `numCols` and a string typed `bucket` function that Spark searches for on the implementing library's side. The name technically doesn't matter for the field but it 100% does for the function.
I hope @huaxingao is right though that we will be able to use Spark to parse more expressions this way though =)
https://github.com/apache/spark/blob/126870eddc57fff7408942897dd59a2c19c91c4a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/V2ExpressionUtils.scala#L90-L100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r936089914
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (checkIfPredicateValid(predicate) == null) {
+ return null;
Review Comment:
But we throw an exception here? Shouldn't the behavior just be to ignore the predicate then?
https://github.com/apache/iceberg/pull/5302/files#diff-d21a6d1f3fb11bbc8d5e9ced54b7fd025e04a70a54c8263ae49313ea3cb274beR84-R85
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#issuecomment-1203078396
also cc @flyrain @szehon-ho @kbendick
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] huaxingao commented on a diff in pull request #5302: Add SparkV2Filters
Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5302:
URL: https://github.com/apache/iceberg/pull/5302#discussion_r936137474
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.expressions.Expressions.and;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.in;
+import static org.apache.iceberg.expressions.Expressions.isNaN;
+import static org.apache.iceberg.expressions.Expressions.isNull;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.not;
+import static org.apache.iceberg.expressions.Expressions.notIn;
+import static org.apache.iceberg.expressions.Expressions.notNull;
+import static org.apache.iceberg.expressions.Expressions.or;
+import static org.apache.iceberg.expressions.Expressions.startsWith;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.util.NaNUtil;
+import org.apache.spark.sql.connector.expressions.LiteralValue;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.And;
+import org.apache.spark.sql.connector.expressions.filter.Not;
+import org.apache.spark.sql.connector.expressions.filter.Or;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+import org.apache.spark.unsafe.types.UTF8String;
+
+public class SparkV2Filters {
+
+ private static final Pattern BACKTICKS_PATTERN = Pattern.compile("([`])(.|$)");
+
+ private SparkV2Filters() {}
+
+ private static final Map<String, Expression.Operation> FILTERS =
+ ImmutableMap.<String, Expression.Operation>builder()
+ .put("ALWAYS_TRUE", Expression.Operation.TRUE)
+ .put("ALWAYS_FALSE", Expression.Operation.FALSE)
+ .put("=", Expression.Operation.EQ)
+ .put("<=>", Expression.Operation.EQ)
+ .put(">", Expression.Operation.GT)
+ .put(">=", Expression.Operation.GT_EQ)
+ .put("<", Expression.Operation.LT)
+ .put("<=", Expression.Operation.LT_EQ)
+ .put("IN", Expression.Operation.IN)
+ .put("IS_NULL", Expression.Operation.IS_NULL)
+ .put("IS_NOT_NULL", Expression.Operation.NOT_NULL)
+ .put("AND", Expression.Operation.AND)
+ .put("OR", Expression.Operation.OR)
+ .put("NOT", Expression.Operation.NOT)
+ .put("STARTS_WITH", Expression.Operation.STARTS_WITH)
+ .build();
+
+ public static Expression convert(Predicate[] predicates) {
+ Expression expression = Expressions.alwaysTrue();
+ for (Predicate predicate : predicates) {
+ Expression converted = convert(predicate);
+ Preconditions.checkArgument(
+ converted != null, "Cannot convert predicate to Iceberg: %s", predicate);
+ expression = Expressions.and(expression, converted);
+ }
+ return expression;
+ }
+
+ @SuppressWarnings({"checkstyle:CyclomaticComplexity", "checkstyle:MethodLength"})
+ public static Expression convert(Predicate predicate) {
+ if (checkIfPredicateValid(predicate) == null) {
+ return null;
Review Comment:
I see what you mean now. Sorry I wasn't realize that you were talking about the convert method that takes the `Predicate` array.
This method `convert(Predicate predicate)` will be used by [`SparkScanBuilder`.`pushPredicates`](https://github.com/apache/iceberg/blob/master/spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java#L111) directly. If the Spark v2 filter can't be converted to Iceberg Expression, we just ignore the filter and not push it down.
The method `convert(Predicate[] predicates)` will be used by [`SparkWriteBuilder`.`overwrite`](https://github.com/apache/iceberg/blob/master/spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkWriteBuilder.java#L122) and [`SparkTable`.`deleteWhere`](https://github.com/apache/iceberg/blob/master/spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java#L311). I follow the same logic as [`SparkFilter.convert(Filter[] filters)`](https://github.com/apache/iceberg/blob/master/spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java#L311). By definition, `overwrite` can only replace data iff all the filters match and `deleteWhere` can only delete rows iff all the filter expressions match. That's why we throw Exception in `convert(Predicate[] predicates)` if any of the filter can't be converted to Iceberg Expression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org