You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ConeyLiu (via GitHub)" <gi...@apache.org> on 2023/07/17 14:11:18 UTC
[GitHub] [iceberg] ConeyLiu opened a new pull request, #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
ConeyLiu opened a new pull request, #8088:
URL: https://github.com/apache/iceberg/pull/8088
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1287039379
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java:
##########
@@ -1304,6 +1313,82 @@ public void testDeleteToCustomWapBranchWithoutWhereClause() throws NoSuchTableEx
});
}
+ @Test
+ public void testDeleteFilterWithSystemFuncOnPartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createPartitionedTable(spark, tableName, "years(ts)", tableProperties);
+
+ String date = "2018-12-21";
+ String remainSql =
+ sqlFormat(
+ "SELECT * FROM %s WHERE %s.system.years(ts) != %s.system.years(date('%s')) ORDER BY id",
+ tableName, catalogName, catalogName, date);
+
+ List<Object[]> expected = sql(remainSql);
+
+ withSQLConf(
+ ImmutableMap.of(SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED, "true"),
+ () -> {
+ String deleteSql =
+ sqlFormat(
+ "DELETE FROM %s WHERE %s.system.years(ts) == %s.system.years(date('%s'))",
+ tableName, catalogName, catalogName, date);
+ Object explain = scalarSql("EXPLAIN EXTENDED %s", deleteSql);
+ // Metadata delete
+ Assertions.assertThat((String) explain).contains("DeleteFromTableWithFilters");
+ Assertions.assertThat((String) explain).contains("[years(ts) = " + years(date));
+
+ sql(deleteSql);
+
+ List<Object[]> actual = sql(remainSql);
+ assertEquals("Results should match", expected, actual);
+ });
+ }
+
+ @Test
+ public void testDeleteFilterWithSystemFuncOnUnpartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createUnpartitionedTable(spark, tableName, tableProperties);
+
+ String date = "2018-12-21";
+ String remainSql =
+ sqlFormat(
+ "SELECT * FROM %s WHERE %s.system.months(ts) = %s.system.months(date('%s')) ORDER BY id",
+ tableName, catalogName, catalogName, date);
+
+ List<Object[]> expected = sql(remainSql);
+
+ withSQLConf(
+ ImmutableMap.of(SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED, "true"),
+ () -> {
+ String deleteSql =
+ sqlFormat(
+ "DELETE FROM %s WHERE %s.system.months(ts) != %s.system.months(date('%s'))",
+ tableName, catalogName, catalogName, date);
+ Object explain = scalarSql("EXPLAIN EXTENDED %s", deleteSql);
+ Assertions.assertThat((String) explain)
+ .contains("Filter NOT (applyfunctionexpression(Wrapper(iceberg.months(timestamp))");
+ Assertions.assertThat((String) explain).contains("[filters=NOT (ts = " + months(date));
Review Comment:
Submitted #8257 for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285270707
##########
spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/SystemFunctionPushDownHelper.java:
##########
@@ -39,6 +56,22 @@ public static void createPartitionedTable(
insertRecords(spark, tableName);
}
+ public static void createPartitionedTable(
+ SparkSession spark, String tableName, String partitionCol, Map<String, String> props) {
+ if (props.isEmpty()) {
+ createPartitionedTable(spark, tableName, partitionCol);
+ } else {
+ String tblProps = Joiner.on("','").withKeyValueSeparator("'='").join(props);
Review Comment:
Iceberg does not needlessly abbreviate words, like "table" as "tbl". It would also be more descriptive to use `propsString` because the significant part is that the properties were converted to a string.
##########
spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/sql/SystemFunctionTestBase.java:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.sql;
+
+import org.apache.iceberg.spark.SparkSQLProperties;
+import org.apache.iceberg.spark.SparkTestBaseWithCatalog;
+import org.junit.Before;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+@RunWith(Parameterized.class)
+public abstract class SystemFunctionTestBase extends SparkTestBaseWithCatalog {
Review Comment:
This should not be needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1283915521
##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetBloomRowGroupFilter.java:
##########
@@ -334,5 +335,10 @@ private <T> boolean shouldRead(
return ROWS_MIGHT_MATCH;
}
}
+
+ @Override
+ public <T> Boolean handleNonReference(Bound<T> term) {
+ return ROWS_MIGHT_MATCH;
Review Comment:
@rdblue this is not for constant folding. The transform function such as `years(ts) = 20` could be pushed to fileformat such as parquet/orc. The default behavior for those non-reference terms is throwing an exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285269918
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java:
##########
@@ -2728,6 +2736,105 @@ public void testMergeToWapBranchWithTableBranchIdentifier() {
branch)));
}
+ @Test
+ public void testMergeFilterWithSystemFunctionsOnPartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createPartitionedTable(spark, tableName, "years(ts)", tableProperties);
+
+ createOrReplaceView(
+ "source",
+ "id BIGINT, ts TIMESTAMP, data STRING",
+ "{ \"id\": 0, \"ts\": \"2023-08-01T09:20:44.294658+00:00\", \"data\": \"new-data-0\"}\n"
+ + "{ \"id\": 5, \"ts\": \"2023-08-02T09:23:44.294658+00:00\", \"data\": \"new-data-5\"}\n");
+
+ String date = "2017-11-22";
+ String resultSql =
+ sqlFormat("SELECT id, ts, IF(id = 0, 'new-data-0', data) FROM %s ORDER BY id", tableName);
+ List<Object[]> expected = sql(resultSql);
+
+ withSQLConf(
+ ImmutableMap.of(SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED, "true"),
+ () -> {
+ String mergeSql =
+ sqlFormat(
+ "MERGE INTO %s AS t USING source AS s "
+ + "ON t.id = s.id AND %s.system.years(t.ts) = %s.system.years(date('%s')) "
+ + "WHEN MATCHED "
+ + " THEN UPDATE SET t.data = s.data "
+ + "WHEN NOT MATCHED AND s.id != 5 "
+ + " THEN INSERT *",
+ tableName, catalogName, catalogName, date);
+ Dataset<Row> df = spark.sql(mergeSql);
+ // Should not contain applyfunction since filter function could be evaluated completely on
+ // Iceberg side and will not return post scan filters
+ Assertions.assertThat(df.queryExecution().sparkPlan().toString())
+ .doesNotContain("Filter (applyfunctionexpression(Wrapper(iceberg.years(timestamp))");
+ Assertions.assertThat(df.queryExecution().sparkPlan().toString())
+ .contains("[filters=ts = " + years(date));
+
+ List<Object[]> actual = sql("SELECT * FROM %s ORDER BY id", tableName);
Review Comment:
This is what the other tests should be doing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1320855027
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceStaticInvoke.scala:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.iceberg.spark.functions.SparkFunctions
+import org.apache.spark.sql.catalyst.SQLConfHelper
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.FILTER
+import org.apache.spark.sql.connector.catalog.functions.ScalarFunction
+import org.apache.spark.sql.types.StructField
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Spark analyzes the Iceberg system function to {@link StaticInvoke} which could not be pushed
+ * down to datasource. This rule will replace {@link StaticInvoke} to
+ * {@link ApplyFunctionExpression} for Iceberg system function in a filter condition.
+ */
+object ReplaceStaticInvoke extends Rule[LogicalPlan] with SQLConfHelper {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.transformWithPruning (
+ _.containsAnyPattern(FILTER)) {
+
+ case filter @ Filter(condition, _) =>
Review Comment:
I think it may be useful to further narrow down the matching expressions by finding conditions that include a subclass of `BinaryComparison`. Then this could match either the left or right side as a `StaticInvoke` and the other side as a foldable. That would reduce false positives.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1320855513
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceStaticInvoke.scala:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.iceberg.spark.functions.SparkFunctions
+import org.apache.spark.sql.catalyst.SQLConfHelper
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.FILTER
+import org.apache.spark.sql.connector.catalog.functions.ScalarFunction
+import org.apache.spark.sql.types.StructField
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Spark analyzes the Iceberg system function to {@link StaticInvoke} which could not be pushed
+ * down to datasource. This rule will replace {@link StaticInvoke} to
+ * {@link ApplyFunctionExpression} for Iceberg system function in a filter condition.
+ */
+object ReplaceStaticInvoke extends Rule[LogicalPlan] with SQLConfHelper {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.transformWithPruning (
+ _.containsAnyPattern(FILTER)) {
+
+ case filter @ Filter(condition, _) =>
+ val newCondition = replaceStaticInvoke(condition)
+ if (newCondition fastEquals condition) {
+ filter
+ } else {
+ filter.copy(condition = newCondition)
+ }
+ }
+
+ private def replaceStaticInvoke(expression: Expression): Expression = expression.transform {
+ case invoke: StaticInvoke =>
Review Comment:
It looks like this could be simpler by moving a few checks into the match expression:
```scala
case invoke: StaticInvoke if invoke.functionName == ScalarFunction.MAGIC_METHOD_NAME && invoke.foldable => ...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1320857552
##########
spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/SystemFunctionPushDownHelper.java:
##########
@@ -39,6 +63,22 @@ public static void createPartitionedTable(
insertRecords(spark, tableName);
}
+ public static void createPartitionedTable(
Review Comment:
I don't think these additional versions are needed since the tests in this PR should not need to exercise different underlying data formats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285270454
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/functions/UnboundFunctionWrapper.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.functions;
+
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.connector.catalog.functions.BoundFunction;
+import org.apache.spark.sql.connector.catalog.functions.ScalarFunction;
+import org.apache.spark.sql.connector.catalog.functions.UnboundFunction;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * An UnboundFunction wrapper that wraps a system function. The bound function by this wrapper will
+ * hide the magic method.
+ */
+public class UnboundFunctionWrapper implements UnboundFunction {
Review Comment:
I don't think it is a good idea to trick Spark, so we should probably not need this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285551672
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/RowLevelCommandScanRelationPushDown.scala:
##########
@@ -122,16 +130,15 @@ object RowLevelCommandScanRelationPushDown extends Rule[LogicalPlan] with Predic
private def pushFilters(
cond: Expression,
scanBuilder: ScanBuilder,
- tableAttrs: Seq[AttributeReference]): (Seq[Filter], Seq[Predicate]) = {
+ tableAttrs: Seq[AttributeReference]): (Either[Seq[Filter], Seq[Predicate]], Seq[Expression]) = {
val tableAttrSet = AttributeSet(tableAttrs)
val filters = splitConjunctivePredicates(cond).filter(_.references.subsetOf(tableAttrSet))
val normalizedFilters = DataSourceStrategy.normalizeExprs(filters, tableAttrs)
val (_, normalizedFiltersWithoutSubquery) =
normalizedFilters.partition(SubqueryExpression.hasSubquery)
- val (pushedFilters, _) = PushDownUtils.pushFilters(scanBuilder, normalizedFiltersWithoutSubquery)
- (pushedFilters.left.getOrElse(Seq.empty), pushedFilters.right.getOrElse(Seq.empty))
+ PushDownUtils.pushFilters(scanBuilder, normalizedFiltersWithoutSubquery)
Review Comment:
Submitted #8249
##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetBloomRowGroupFilter.java:
##########
@@ -334,5 +335,10 @@ private <T> boolean shouldRead(
return ROWS_MIGHT_MATCH;
}
}
+
+ @Override
+ public <T> Boolean handleNonReference(Bound<T> term) {
+ return ROWS_MIGHT_MATCH;
Review Comment:
Submitted #8243 for parquet and #8244 for orc. Will rebase after they are merged. Keep here for CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285977810
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java:
##########
@@ -1304,6 +1313,82 @@ public void testDeleteToCustomWapBranchWithoutWhereClause() throws NoSuchTableEx
});
}
+ @Test
+ public void testDeleteFilterWithSystemFuncOnPartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createPartitionedTable(spark, tableName, "years(ts)", tableProperties);
+
+ String date = "2018-12-21";
+ String remainSql =
+ sqlFormat(
+ "SELECT * FROM %s WHERE %s.system.years(ts) != %s.system.years(date('%s')) ORDER BY id",
+ tableName, catalogName, catalogName, date);
+
+ List<Object[]> expected = sql(remainSql);
+
+ withSQLConf(
+ ImmutableMap.of(SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED, "true"),
+ () -> {
+ String deleteSql =
+ sqlFormat(
+ "DELETE FROM %s WHERE %s.system.years(ts) == %s.system.years(date('%s'))",
+ tableName, catalogName, catalogName, date);
+ Object explain = scalarSql("EXPLAIN EXTENDED %s", deleteSql);
+ // Metadata delete
+ Assertions.assertThat((String) explain).contains("DeleteFromTableWithFilters");
+ Assertions.assertThat((String) explain).contains("[years(ts) = " + years(date));
+
+ sql(deleteSql);
+
+ List<Object[]> actual = sql(remainSql);
Review Comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1320857458
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestSystemFunctionPushDownDQL.java:
##########
@@ -0,0 +1,338 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.extensions;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT;
+import static org.apache.iceberg.expressions.Expressions.bucket;
+import static org.apache.iceberg.expressions.Expressions.day;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.hour;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.month;
+import static org.apache.iceberg.expressions.Expressions.notEqual;
+import static org.apache.iceberg.expressions.Expressions.truncate;
+import static org.apache.iceberg.expressions.Expressions.year;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.STRUCT;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.createPartitionedTable;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.createUnpartitionedTable;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.timestampStrToDayOrdinal;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.timestampStrToHourOrdinal;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.timestampStrToMonthOrdinal;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.timestampStrToYearOrdinal;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.expressions.ExpressionUtil;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.spark.SparkCatalogConfig;
+import org.apache.iceberg.spark.source.PlanUtils;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression;
+import org.apache.spark.sql.catalyst.expressions.Expression;
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke;
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan;
+import org.assertj.core.api.Assertions;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runners.Parameterized;
+
+public class TestSystemFunctionPushDownDQL extends SparkExtensionsTestBase {
+ private final String fileFormat;
+
+ public TestSystemFunctionPushDownDQL(
+ String catalogName, String implementation, Map<String, String> config, String fileFormat) {
+ super(catalogName, implementation, config);
+ this.fileFormat = fileFormat;
+ }
+
+ @Parameterized.Parameters(
+ name = "catalogName = {0}, implementation = {1}, config = {2}, fileFormat = {3}")
+ public static Object[][] parameters() {
+ return new Object[][] {
+ {
+ SparkCatalogConfig.HIVE.catalogName(),
+ SparkCatalogConfig.HIVE.implementation(),
+ SparkCatalogConfig.HIVE.properties(),
+ "parquet"
Review Comment:
Testing pushdown should not depend on the file format. Testing the file format filters would require it, but this should be the same regardless of format.
Can you remove these parameters and use a single run?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1286962161
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java:
##########
@@ -2728,6 +2736,105 @@ public void testMergeToWapBranchWithTableBranchIdentifier() {
branch)));
}
+ @Test
+ public void testMergeFilterWithSystemFunctionsOnPartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createPartitionedTable(spark, tableName, "years(ts)", tableProperties);
+
+ createOrReplaceView(
+ "source",
+ "id BIGINT, ts TIMESTAMP, data STRING",
+ "{ \"id\": 0, \"ts\": \"2023-08-01T09:20:44.294658+00:00\", \"data\": \"new-data-0\"}\n"
+ + "{ \"id\": 5, \"ts\": \"2023-08-02T09:23:44.294658+00:00\", \"data\": \"new-data-5\"}\n");
+
+ String date = "2017-11-22";
+ String resultSql =
+ sqlFormat("SELECT id, ts, IF(id = 0, 'new-data-0', data) FROM %s ORDER BY id", tableName);
+ List<Object[]> expected = sql(resultSql);
+
+ withSQLConf(
+ ImmutableMap.of(SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED, "true"),
+ () -> {
+ String mergeSql =
+ sqlFormat(
+ "MERGE INTO %s AS t USING source AS s "
+ + "ON t.id = s.id AND %s.system.years(t.ts) = %s.system.years(date('%s')) "
+ + "WHEN MATCHED "
+ + " THEN UPDATE SET t.data = s.data "
+ + "WHEN NOT MATCHED AND s.id != 5 "
+ + " THEN INSERT *",
+ tableName, catalogName, catalogName, date);
+ Dataset<Row> df = spark.sql(mergeSql);
+ // Should not contain applyfunction since filter function could be evaluated completely on
+ // Iceberg side and will not return post scan filters
+ Assertions.assertThat(df.queryExecution().sparkPlan().toString())
+ .doesNotContain("Filter (applyfunctionexpression(Wrapper(iceberg.years(timestamp))");
Review Comment:
Row-level commands UTs updated as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285269448
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java:
##########
@@ -2728,6 +2736,105 @@ public void testMergeToWapBranchWithTableBranchIdentifier() {
branch)));
}
+ @Test
+ public void testMergeFilterWithSystemFunctionsOnPartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createPartitionedTable(spark, tableName, "years(ts)", tableProperties);
+
+ createOrReplaceView(
+ "source",
+ "id BIGINT, ts TIMESTAMP, data STRING",
+ "{ \"id\": 0, \"ts\": \"2023-08-01T09:20:44.294658+00:00\", \"data\": \"new-data-0\"}\n"
+ + "{ \"id\": 5, \"ts\": \"2023-08-02T09:23:44.294658+00:00\", \"data\": \"new-data-5\"}\n");
+
+ String date = "2017-11-22";
+ String resultSql =
+ sqlFormat("SELECT id, ts, IF(id = 0, 'new-data-0', data) FROM %s ORDER BY id", tableName);
+ List<Object[]> expected = sql(resultSql);
+
+ withSQLConf(
+ ImmutableMap.of(SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED, "true"),
+ () -> {
+ String mergeSql =
+ sqlFormat(
+ "MERGE INTO %s AS t USING source AS s "
+ + "ON t.id = s.id AND %s.system.years(t.ts) = %s.system.years(date('%s')) "
+ + "WHEN MATCHED "
+ + " THEN UPDATE SET t.data = s.data "
+ + "WHEN NOT MATCHED AND s.id != 5 "
+ + " THEN INSERT *",
+ tableName, catalogName, catalogName, date);
+ Dataset<Row> df = spark.sql(mergeSql);
+ // Should not contain applyfunction since filter function could be evaluated completely on
+ // Iceberg side and will not return post scan filters
+ Assertions.assertThat(df.queryExecution().sparkPlan().toString())
+ .doesNotContain("Filter (applyfunctionexpression(Wrapper(iceberg.years(timestamp))");
Review Comment:
Using `doesNotContain` is a very brittle assertion because it could be true if Spark changes the format of its string representation. Can you instead find a way to test more directly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285271458
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/FoldConstantSystemFunction.scala:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.iceberg.spark.functions.UnboundFunctionWrapper.SystemFunctionWrapper
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression
+import org.apache.spark.sql.catalyst.expressions.EmptyRow
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.plans.logical.LocalRelation
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * Folding those system function that applies a constant value.
+ */
+object FoldConstantSystemFunction extends Rule[LogicalPlan] {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+ case logicalPlan: LogicalPlan =>
+ logicalPlan transformExpressions {
+ case applyFunc: ApplyFunctionExpression
+ if applyFunc.function.isInstanceOf[SystemFunctionWrapper[_]] =>
+ val fakeProjectList = applyFunc.children.zipWithIndex.map(pair => Alias(pair._1, s"fake_${pair._2}")())
Review Comment:
Why is the alias needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1282922082
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/RowLevelCommandScanRelationPushDown.scala:
##########
@@ -122,16 +130,15 @@ object RowLevelCommandScanRelationPushDown extends Rule[LogicalPlan] with Predic
private def pushFilters(
cond: Expression,
scanBuilder: ScanBuilder,
- tableAttrs: Seq[AttributeReference]): (Seq[Filter], Seq[Predicate]) = {
+ tableAttrs: Seq[AttributeReference]): (Either[Seq[Filter], Seq[Predicate]], Seq[Expression]) = {
val tableAttrSet = AttributeSet(tableAttrs)
val filters = splitConjunctivePredicates(cond).filter(_.references.subsetOf(tableAttrSet))
val normalizedFilters = DataSourceStrategy.normalizeExprs(filters, tableAttrs)
val (_, normalizedFiltersWithoutSubquery) =
normalizedFilters.partition(SubqueryExpression.hasSubquery)
- val (pushedFilters, _) = PushDownUtils.pushFilters(scanBuilder, normalizedFiltersWithoutSubquery)
- (pushedFilters.left.getOrElse(Seq.empty), pushedFilters.right.getOrElse(Seq.empty))
+ PushDownUtils.pushFilters(scanBuilder, normalizedFiltersWithoutSubquery)
Review Comment:
The returning pair should be `(pushedFilters, unpushedFilters)`.
```scala
// PushDownUtils
case r: SupportsPushDownFilters =>
...
(Left(r.pushedFilters()), (postScanFilters ++ untranslatableExprs).toSeq)
case r: SupportsPushDownV2Filters =>
...
(Right(r.pushedPredicates), (postScanFilters ++ untranslatableExprs).toSeq)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1282916244
##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetBloomRowGroupFilter.java:
##########
@@ -334,5 +335,10 @@ private <T> boolean shouldRead(
return ROWS_MIGHT_MATCH;
}
}
+
+ @Override
+ public <T> Boolean handleNonReference(Bound<T> term) {
+ return ROWS_MIGHT_MATCH;
Review Comment:
Throw exceptions by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1282922082
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/RowLevelCommandScanRelationPushDown.scala:
##########
@@ -122,16 +130,15 @@ object RowLevelCommandScanRelationPushDown extends Rule[LogicalPlan] with Predic
private def pushFilters(
cond: Expression,
scanBuilder: ScanBuilder,
- tableAttrs: Seq[AttributeReference]): (Seq[Filter], Seq[Predicate]) = {
+ tableAttrs: Seq[AttributeReference]): (Either[Seq[Filter], Seq[Predicate]], Seq[Expression]) = {
val tableAttrSet = AttributeSet(tableAttrs)
val filters = splitConjunctivePredicates(cond).filter(_.references.subsetOf(tableAttrSet))
val normalizedFilters = DataSourceStrategy.normalizeExprs(filters, tableAttrs)
val (_, normalizedFiltersWithoutSubquery) =
normalizedFilters.partition(SubqueryExpression.hasSubquery)
- val (pushedFilters, _) = PushDownUtils.pushFilters(scanBuilder, normalizedFiltersWithoutSubquery)
- (pushedFilters.left.getOrElse(Seq.empty), pushedFilters.right.getOrElse(Seq.empty))
+ PushDownUtils.pushFilters(scanBuilder, normalizedFiltersWithoutSubquery)
Review Comment:
The returning pair of `PushDownUtils.pushFilters` should be `(pushedFilters, unpushedFilters)`.
```scala
// PushDownUtils
case r: SupportsPushDownFilters =>
...
(Left(r.pushedFilters()), (postScanFilters ++ untranslatableExprs).toSeq)
case r: SupportsPushDownV2Filters =>
...
(Right(r.pushedPredicates), (postScanFilters ++ untranslatableExprs).toSeq)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285622310
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java:
##########
@@ -63,4 +63,9 @@ private SparkSQLProperties() {}
// Controls the WAP branch used for write-audit-publish workflow.
// When set, new snapshots will be committed to this branch.
public static final String WAP_BRANCH = "spark.wap.branch";
+
+ // Controls whether to push down Iceberg system function
+ public static final String SYSTEM_FUNC_PUSH_DOWN_ENABLED =
Review Comment:
The system function could be analyzed into `StaticInvoke` or `ApplyFunctionExpression`. However, it will be analyzed into `StaticInvoke` if it has a [magic method](https://github.com/apache/spark/blob/f139733b92d421233a3fd35374236ef084dfd10d/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/functions/ScalarFunction.java#L35).
`StaticInvoke` can not be translated into a data source filter while `ApplyFunctionExpression` could. The `ApplyFunctionExpression` is for a user-defined function and will be translated into `UserDefinedScalarFunc` in filter pushing down. That's the reason I add a wrapper class to hide the magic method for the system functions.
The config here is because `ApplyFunctionExpression` can not leverage WholeStageCodegen which could have a slight performance decrease in some cases.
The workaround solution is not the best one. The best choice is we patch spark upstream to support translating `StaticInvoke` to the data source filter or leveraging the magic method in `ApplyFunctionExpression` for some cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285271442
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/FoldConstantSystemFunction.scala:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.iceberg.spark.functions.UnboundFunctionWrapper.SystemFunctionWrapper
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression
+import org.apache.spark.sql.catalyst.expressions.EmptyRow
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.plans.logical.LocalRelation
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * Folding those system function that applies a constant value.
+ */
+object FoldConstantSystemFunction extends Rule[LogicalPlan] {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+ case logicalPlan: LogicalPlan =>
+ logicalPlan transformExpressions {
+ case applyFunc: ApplyFunctionExpression
+ if applyFunc.function.isInstanceOf[SystemFunctionWrapper[_]] =>
+ val fakeProjectList = applyFunc.children.zipWithIndex.map(pair => Alias(pair._1, s"fake_${pair._2}")())
+ val fakePlan = Project(fakeProjectList, LocalRelation(Nil))
+ val foldedExprs =
+ ConstantFolding.apply(fakePlan).asInstanceOf[Project].projectList.map(_.asInstanceOf[Alias].child)
Review Comment:
Why is this fake plan needed? Wouldn't Spark already call `ConstantFolding` on both `ApplyFunctionExpression` and `StaticInvoke`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285268601
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java:
##########
@@ -1304,6 +1313,82 @@ public void testDeleteToCustomWapBranchWithoutWhereClause() throws NoSuchTableEx
});
}
+ @Test
+ public void testDeleteFilterWithSystemFuncOnPartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createPartitionedTable(spark, tableName, "years(ts)", tableProperties);
+
+ String date = "2018-12-21";
+ String remainSql =
+ sqlFormat(
+ "SELECT * FROM %s WHERE %s.system.years(ts) != %s.system.years(date('%s')) ORDER BY id",
Review Comment:
This should not rely on constant folding to produce the expected rows. If it does, then problems could affect both. Instead, this should calculate the value directly and embed it in the SQL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285270223
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java:
##########
@@ -63,4 +63,9 @@ private SparkSQLProperties() {}
// Controls the WAP branch used for write-audit-publish workflow.
// When set, new snapshots will be committed to this branch.
public static final String WAP_BRANCH = "spark.wap.branch";
+
+ // Controls whether to push down Iceberg system function
+ public static final String SYSTEM_FUNC_PUSH_DOWN_ENABLED =
Review Comment:
Ah, I see from the file below.
Looks like folding constant expressions relies on `ApplyFunctionExpression` rather than `StaticInvoke`. To get a plan with `ApplyFunctionExpression`, this inserts a wrapper. I don't think that's a good idea, no is having a property to enable this.
Why can't constant folding work with `StaticInvoke`? Why is constant folding in Iceberg rather than Spark necessary in the first place?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1283452305
##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetBloomRowGroupFilter.java:
##########
@@ -334,5 +335,10 @@ private <T> boolean shouldRead(
return ROWS_MIGHT_MATCH;
}
}
+
+ @Override
+ public <T> Boolean handleNonReference(Bound<T> term) {
+ return ROWS_MIGHT_MATCH;
Review Comment:
Why does this need to change? Changing the Filter implementations just to implement constant folding in Spark in a big red flag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1286439148
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestSystemFunctionPushDownDQL.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.extensions;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.createPartitionedTable;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.createUnpartitionedTable;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.days;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.hours;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.months;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.years;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.spark.SparkCatalogConfig;
+import org.apache.iceberg.spark.SparkSQLProperties;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.assertj.core.api.Assertions;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runners.Parameterized;
+
+public class TestSystemFunctionPushDownDQL extends SparkExtensionsTestBase {
+ private final String fileFormat;
+
+ public TestSystemFunctionPushDownDQL(
Review Comment:
Let's see what gets included in #8243.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1286439607
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java:
##########
@@ -63,4 +63,9 @@ private SparkSQLProperties() {}
// Controls the WAP branch used for write-audit-publish workflow.
// When set, new snapshots will be committed to this branch.
public static final String WAP_BRANCH = "spark.wap.branch";
+
+ // Controls whether to push down Iceberg system function
+ public static final String SYSTEM_FUNC_PUSH_DOWN_ENABLED =
Review Comment:
I'm not following. Is this PR intended to fix pushdown or constant folding? It seems like it should be fixing constant folding because pushdown is working just fine with `StaticInvoke` after the last PR isn't it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1321447450
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/source/PlanUtils.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan;
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation;
+import scala.PartialFunction;
+import scala.collection.JavaConverters;
+import scala.collection.Seq;
+
+public class PlanUtils {
+ private PlanUtils() {}
+
+ public static List<Expression> getScanPushDownFilters(LogicalPlan logicalPlan) {
Review Comment:
Renamed to `collectPushDownFilters`.
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/source/PlanUtils.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan;
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation;
+import scala.PartialFunction;
+import scala.collection.JavaConverters;
+import scala.collection.Seq;
+
+public class PlanUtils {
+ private PlanUtils() {}
+
+ public static List<Expression> getScanPushDownFilters(LogicalPlan logicalPlan) {
+ return JavaConverters.asJavaCollection(logicalPlan.collectLeaves()).stream()
+ .flatMap(
+ plan -> {
+ if (!(plan instanceof DataSourceV2ScanRelation)) {
+ return Stream.empty();
+ }
+
+ DataSourceV2ScanRelation scanRelation = (DataSourceV2ScanRelation) plan;
+ if (!(scanRelation.scan() instanceof SparkBatchQueryScan)) {
+ return Stream.empty();
+ }
+
+ SparkBatchQueryScan batchQueryScan = (SparkBatchQueryScan) scanRelation.scan();
+ return batchQueryScan.filterExpressions().stream();
+ })
+ .collect(Collectors.toList());
+ }
+
+ public static List<org.apache.spark.sql.catalyst.expressions.Expression> collectSparkExpressions(
+ LogicalPlan logicalPlan,
+ Function<org.apache.spark.sql.catalyst.expressions.Expression, Boolean> filterFunction) {
Review Comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1320857230
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/source/PlanUtils.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan;
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation;
+import scala.PartialFunction;
+import scala.collection.JavaConverters;
+import scala.collection.Seq;
+
+public class PlanUtils {
+ private PlanUtils() {}
+
+ public static List<Expression> getScanPushDownFilters(LogicalPlan logicalPlan) {
+ return JavaConverters.asJavaCollection(logicalPlan.collectLeaves()).stream()
+ .flatMap(
+ plan -> {
+ if (!(plan instanceof DataSourceV2ScanRelation)) {
+ return Stream.empty();
+ }
+
+ DataSourceV2ScanRelation scanRelation = (DataSourceV2ScanRelation) plan;
+ if (!(scanRelation.scan() instanceof SparkBatchQueryScan)) {
+ return Stream.empty();
+ }
+
+ SparkBatchQueryScan batchQueryScan = (SparkBatchQueryScan) scanRelation.scan();
+ return batchQueryScan.filterExpressions().stream();
+ })
+ .collect(Collectors.toList());
+ }
+
+ public static List<org.apache.spark.sql.catalyst.expressions.Expression> collectSparkExpressions(
+ LogicalPlan logicalPlan,
+ Function<org.apache.spark.sql.catalyst.expressions.Expression, Boolean> filterFunction) {
Review Comment:
It looks like this should be a `Predicate` instead of a `Function`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1284508686
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteStaticInvoke.scala:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.iceberg.spark.SparkSQLProperties
+import org.apache.iceberg.spark.functions.SparkFunctions
+import org.apache.iceberg.spark.source.SparkTable
+import org.apache.spark.sql.catalyst.SQLConfHelper
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke
+import org.apache.spark.sql.catalyst.planning.RewrittenRowLevelCommand
+import org.apache.spark.sql.catalyst.planning.ScanOperation
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.NoStatsUnaryNode
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceIcebergData
+import org.apache.spark.sql.catalyst.plans.logical.RowLevelCommand
+import org.apache.spark.sql.catalyst.plans.logical.WriteDelta
+import org.apache.spark.sql.catalyst.plans.logical.WriteIcebergDelta
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.connector.catalog.Table
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation
+import org.apache.spark.sql.execution.datasources.v2.UnplannedGroupBasedMergeOperation
+import org.apache.spark.sql.types.AbstractDataType
+import org.apache.spark.sql.types.DataType
+import scala.annotation.tailrec
+
+/**
+ * Spark analyzes the Iceberg system function to {@link StaticInvoke} which could not be pushed
+ * down to datasource. This rule will replace {@link StaticInvoke} to
+ * {@link ApplyFunctionExpression} for Iceberg system function in a filter condition.
+ */
+case object RewriteStaticInvoke extends Rule[LogicalPlan] with SQLConfHelper {
Review Comment:
Another choice here is using a wrapper class to wrap the system function while not implementing the `magic method function` when enabled the system function push down. Then the system function will be analyzed into `ApplyFunctionExpression `, so we don't need to rewrite the filter expression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1294047348
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java:
##########
@@ -63,4 +63,9 @@ private SparkSQLProperties() {}
// Controls the WAP branch used for write-audit-publish workflow.
// When set, new snapshots will be committed to this branch.
public static final String WAP_BRANCH = "spark.wap.branch";
+
+ // Controls whether to push down Iceberg system function
+ public static final String SYSTEM_FUNC_PUSH_DOWN_ENABLED =
Review Comment:
I went through some of those old commits over the weekend and I think that converting `StaticInvoke` when necessary is a good idea. I think we may also be able to simplify some of the rules, since the pushdown logic needs additional handling for reasons that don't necessarily limit rules for converting expressions. I'd like to see if we can simplify the rules to not have quite so many special cases.
And I also think we don't need to use the special matchers since we don't have special logic. We can probably just use the logical plans directly if need to support specific plan nodes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285228266
##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetDictionaryRowGroupFilter.java:
##########
@@ -463,6 +464,11 @@ private <T> Set<T> dict(int id, Comparator<T> comparator) {
return dictSet;
}
+
+ @Override
+ public <T> Boolean handleNonReference(Bound<T> term) {
Review Comment:
Same for parquet too. The reason is that we support push-down transform functions now. However, the file format can't handle such filters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1286961528
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java:
##########
@@ -1304,6 +1313,82 @@ public void testDeleteToCustomWapBranchWithoutWhereClause() throws NoSuchTableEx
});
}
+ @Test
+ public void testDeleteFilterWithSystemFuncOnPartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createPartitionedTable(spark, tableName, "years(ts)", tableProperties);
+
+ String date = "2018-12-21";
+ String remainSql =
+ sqlFormat(
+ "SELECT * FROM %s WHERE %s.system.years(ts) != %s.system.years(date('%s')) ORDER BY id",
Review Comment:
OK, updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285557036
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java:
##########
@@ -1304,6 +1313,82 @@ public void testDeleteToCustomWapBranchWithoutWhereClause() throws NoSuchTableEx
});
}
+ @Test
+ public void testDeleteFilterWithSystemFuncOnPartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createPartitionedTable(spark, tableName, "years(ts)", tableProperties);
+
+ String date = "2018-12-21";
+ String remainSql =
+ sqlFormat(
+ "SELECT * FROM %s WHERE %s.system.years(ts) != %s.system.years(date('%s')) ORDER BY id",
Review Comment:
Here, I want to test the constant system function could fold as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1282955480
##########
orc/src/main/java/org/apache/iceberg/orc/ExpressionToSearchArgument.java:
##########
@@ -270,7 +271,8 @@ public <T> Action notStartsWith(Bound<T> expr, Literal<T> lit) {
@Override
public <T> Action predicate(BoundPredicate<T> pred) {
- if (UNSUPPORTED_TYPES.contains(pred.ref().type().typeId())) {
+ if (UNSUPPORTED_TYPES.contains(pred.ref().type().typeId())
+ || !(pred.term() instanceof BoundReference)) {
Review Comment:
The reason is that we support push-down transform functions now. However, the file format can't handle such filters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285269492
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java:
##########
@@ -2728,6 +2736,105 @@ public void testMergeToWapBranchWithTableBranchIdentifier() {
branch)));
}
+ @Test
+ public void testMergeFilterWithSystemFunctionsOnPartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createPartitionedTable(spark, tableName, "years(ts)", tableProperties);
+
+ createOrReplaceView(
+ "source",
+ "id BIGINT, ts TIMESTAMP, data STRING",
+ "{ \"id\": 0, \"ts\": \"2023-08-01T09:20:44.294658+00:00\", \"data\": \"new-data-0\"}\n"
+ + "{ \"id\": 5, \"ts\": \"2023-08-02T09:23:44.294658+00:00\", \"data\": \"new-data-5\"}\n");
+
+ String date = "2017-11-22";
+ String resultSql =
+ sqlFormat("SELECT id, ts, IF(id = 0, 'new-data-0', data) FROM %s ORDER BY id", tableName);
+ List<Object[]> expected = sql(resultSql);
+
+ withSQLConf(
+ ImmutableMap.of(SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED, "true"),
+ () -> {
+ String mergeSql =
+ sqlFormat(
+ "MERGE INTO %s AS t USING source AS s "
+ + "ON t.id = s.id AND %s.system.years(t.ts) = %s.system.years(date('%s')) "
+ + "WHEN MATCHED "
+ + " THEN UPDATE SET t.data = s.data "
+ + "WHEN NOT MATCHED AND s.id != 5 "
+ + " THEN INSERT *",
+ tableName, catalogName, catalogName, date);
+ Dataset<Row> df = spark.sql(mergeSql);
+ // Should not contain applyfunction since filter function could be evaluated completely on
+ // Iceberg side and will not return post scan filters
+ Assertions.assertThat(df.queryExecution().sparkPlan().toString())
+ .doesNotContain("Filter (applyfunctionexpression(Wrapper(iceberg.years(timestamp))");
+ Assertions.assertThat(df.queryExecution().sparkPlan().toString())
+ .contains("[filters=ts = " + years(date));
Review Comment:
This also looks incorrect because it drops the transform on `ts`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1293647573
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java:
##########
@@ -1304,6 +1322,97 @@ public void testDeleteToCustomWapBranchWithoutWhereClause() throws NoSuchTableEx
});
}
+ @Test
+ public void testDeleteFilterWithSystemFuncOnPartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createPartitionedTable(spark, tableName, "years(ts)", tableProperties);
+
+ int targetYears = years("2018-12-21");
Review Comment:
Updated based on comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1320858065
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestSystemFunctionPushDownDQL.java:
##########
@@ -0,0 +1,338 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.extensions;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT;
+import static org.apache.iceberg.expressions.Expressions.bucket;
+import static org.apache.iceberg.expressions.Expressions.day;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.hour;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.month;
+import static org.apache.iceberg.expressions.Expressions.notEqual;
+import static org.apache.iceberg.expressions.Expressions.truncate;
+import static org.apache.iceberg.expressions.Expressions.year;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.STRUCT;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.createPartitionedTable;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.createUnpartitionedTable;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.timestampStrToDayOrdinal;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.timestampStrToHourOrdinal;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.timestampStrToMonthOrdinal;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.timestampStrToYearOrdinal;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.expressions.ExpressionUtil;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.spark.SparkCatalogConfig;
+import org.apache.iceberg.spark.source.PlanUtils;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression;
+import org.apache.spark.sql.catalyst.expressions.Expression;
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke;
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan;
+import org.assertj.core.api.Assertions;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runners.Parameterized;
+
+public class TestSystemFunctionPushDownDQL extends SparkExtensionsTestBase {
+ private final String fileFormat;
+
+ public TestSystemFunctionPushDownDQL(
+ String catalogName, String implementation, Map<String, String> config, String fileFormat) {
+ super(catalogName, implementation, config);
+ this.fileFormat = fileFormat;
+ }
+
+ @Parameterized.Parameters(
+ name = "catalogName = {0}, implementation = {1}, config = {2}, fileFormat = {3}")
+ public static Object[][] parameters() {
+ return new Object[][] {
+ {
+ SparkCatalogConfig.HIVE.catalogName(),
+ SparkCatalogConfig.HIVE.implementation(),
+ SparkCatalogConfig.HIVE.properties(),
+ "parquet"
+ },
+ {
+ SparkCatalogConfig.HIVE.catalogName(),
+ SparkCatalogConfig.HIVE.implementation(),
+ SparkCatalogConfig.HIVE.properties(),
+ "orc"
+ },
+ {
+ SparkCatalogConfig.HIVE.catalogName(),
+ SparkCatalogConfig.HIVE.implementation(),
+ SparkCatalogConfig.HIVE.properties(),
+ "avro"
+ },
+ };
+ }
+
+ @Before
+ public void before() {
+ sql("USE %s", catalogName);
+ }
+
+ @After
+ public void removeTables() {
+ sql("DROP TABLE IF EXISTS %s", tableName);
+ }
+
+ @Test
+ public void testYearsFunctionOnUnpartitionedTable() {
+ createUnpartitionedTable(spark, tableName, ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testYearsFunction(false);
+ }
+
+ @Test
+ public void testYearsFunctionOnPartitionedTable() {
+ createPartitionedTable(
+ spark, tableName, "years(ts)", ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testYearsFunction(true);
+ }
+
+ private void testYearsFunction(boolean partitioned) {
+ int targetYears = timestampStrToYearOrdinal("2017-11-22T00:00:00.000000+00:00");
+ String query =
+ String.format(
+ "SELECT * FROM %s WHERE system.years(ts) = %s ORDER BY id", tableName, targetYears);
+
+ Dataset<Row> df = spark.sql(query);
+ LogicalPlan optimizedPlan = df.queryExecution().optimizedPlan();
+
+ checkExpressions(optimizedPlan, partitioned, "years");
+ checkPushedFilters(optimizedPlan, equal(year("ts"), targetYears));
+
+ List<Object[]> actual = rowsToJava(df.collectAsList());
+ Assertions.assertThat(actual.size()).isEqualTo(5);
+ }
+
+ @Test
+ public void testMonthsFunctionOnUnpartitionedTable() {
+ createUnpartitionedTable(spark, tableName, ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testMonthsFunction(false);
+ }
+
+ @Test
+ public void testMonthsFunctionOnPartitionedTable() {
+ createPartitionedTable(
+ spark, tableName, "months(ts)", ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testMonthsFunction(true);
+ }
+
+ private void testMonthsFunction(boolean partitioned) {
+ int targetMonths = timestampStrToMonthOrdinal("2017-11-22T00:00:00.000000+00:00");
+ String query =
+ String.format(
+ "SELECT * FROM %s WHERE system.months(ts) > %s ORDER BY id", tableName, targetMonths);
+
+ Dataset<Row> df = spark.sql(query);
+ LogicalPlan optimizedPlan = df.queryExecution().optimizedPlan();
+
+ checkExpressions(optimizedPlan, partitioned, "months");
+ checkPushedFilters(optimizedPlan, greaterThan(month("ts"), targetMonths));
+
+ List<Object[]> actual = rowsToJava(df.collectAsList());
+ Assertions.assertThat(actual.size()).isEqualTo(5);
+ }
+
+ @Test
+ public void testDaysFunctionOnUnpartitionedTable() {
+ createUnpartitionedTable(spark, tableName, ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testDaysFunction(false);
+ }
+
+ @Test
+ public void testDaysFunctionOnPartitionedTable() {
+ createPartitionedTable(
+ spark, tableName, "days(ts)", ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testDaysFunction(true);
+ }
+
+ private void testDaysFunction(boolean partitioned) {
+ String timestamp = "2018-11-20T00:00:00.000000+00:00";
+ int targetDays = timestampStrToDayOrdinal(timestamp);
+ String query =
+ String.format(
+ "SELECT * FROM %s WHERE system.days(ts) < date('%s') ORDER BY id",
+ tableName, timestamp);
+
+ Dataset<Row> df = spark.sql(query);
+ LogicalPlan optimizedPlan = df.queryExecution().optimizedPlan();
+
+ checkExpressions(optimizedPlan, partitioned, "days");
+ checkPushedFilters(optimizedPlan, lessThan(day("ts"), targetDays));
+
+ List<Object[]> actual = rowsToJava(df.collectAsList());
+ Assertions.assertThat(actual.size()).isEqualTo(5);
+ }
+
+ @Test
+ public void testHoursFunctionOnUnpartitionedTable() {
+ createUnpartitionedTable(spark, tableName, ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testHoursFunction(false);
+ }
+
+ @Test
+ public void testHoursFunctionOnPartitionedTable() {
+ createPartitionedTable(
+ spark, tableName, "hours(ts)", ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testHoursFunction(true);
+ }
+
+ private void testHoursFunction(boolean partitioned) {
+ int targetHours = timestampStrToHourOrdinal("2017-11-22T06:02:09.243857+00:00");
+ String query =
+ String.format(
+ "SELECT * FROM %s WHERE system.hours(ts) >= %s ORDER BY id", tableName, targetHours);
+
+ Dataset<Row> df = spark.sql(query);
+ LogicalPlan optimizedPlan = df.queryExecution().optimizedPlan();
+
+ checkExpressions(optimizedPlan, partitioned, "hours");
+ checkPushedFilters(optimizedPlan, greaterThanOrEqual(hour("ts"), targetHours));
+
+ List<Object[]> actual = rowsToJava(df.collectAsList());
+ Assertions.assertThat(actual.size()).isEqualTo(8);
+ }
+
+ @Test
+ public void testBucketLongFunctionOnUnpartitionedTable() {
+ createUnpartitionedTable(spark, tableName, ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testBucketLongFunction(false);
+ }
+
+ @Test
+ public void testBucketLongFunctionOnPartitionedTable() {
+ createPartitionedTable(
+ spark, tableName, "bucket(5, id)", ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testBucketLongFunction(true);
+ }
+
+ private void testBucketLongFunction(boolean partitioned) {
+ int target = 2;
+ String query =
+ String.format(
+ "SELECT * FROM %s WHERE system.bucket(5, id) <= %s ORDER BY id", tableName, target);
+
+ Dataset<Row> df = spark.sql(query);
+ LogicalPlan optimizedPlan = df.queryExecution().optimizedPlan();
+
+ checkExpressions(optimizedPlan, partitioned, "bucket");
+ checkPushedFilters(optimizedPlan, lessThanOrEqual(bucket("id", 5), target));
+
+ List<Object[]> actual = rowsToJava(df.collectAsList());
+ Assertions.assertThat(actual.size()).isEqualTo(5);
+ }
+
+ @Test
+ public void testBucketStringFunctionOnUnpartitionedTable() {
+ createUnpartitionedTable(spark, tableName, ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testBucketStringFunction(false);
+ }
+
+ @Test
+ public void testBucketStringFunctionOnPartitionedTable() {
+ createPartitionedTable(
+ spark, tableName, "bucket(5, data)", ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testBucketStringFunction(true);
+ }
+
+ private void testBucketStringFunction(boolean partitioned) {
+ int target = 2;
+ String query =
+ String.format(
+ "SELECT * FROM %s WHERE system.bucket(5, data) != %s ORDER BY id", tableName, target);
+
+ Dataset<Row> df = spark.sql(query);
+ LogicalPlan optimizedPlan = df.queryExecution().optimizedPlan();
+
+ checkExpressions(optimizedPlan, partitioned, "bucket");
+ checkPushedFilters(optimizedPlan, notEqual(bucket("data", 5), target));
+
+ List<Object[]> actual = rowsToJava(df.collectAsList());
+ Assertions.assertThat(actual.size()).isEqualTo(8);
+ }
+
+ @Test
+ public void testTruncateFunctionOnUnpartitionedTable() {
+ createUnpartitionedTable(spark, tableName, ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testTruncateFunction(false);
+ }
+
+ @Test
+ public void testTruncateFunctionOnPartitionedTable() {
+ createPartitionedTable(
+ spark, tableName, "truncate(4, data)", ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testTruncateFunction(true);
+ }
+
+ private void testTruncateFunction(boolean partitioned) {
+ String target = "data";
+ String query =
+ String.format(
+ "SELECT * FROM %s WHERE system.truncate(4, data) = '%s' ORDER BY id",
+ tableName, target);
+
+ Dataset<Row> df = spark.sql(query);
+ LogicalPlan optimizedPlan = df.queryExecution().optimizedPlan();
+
+ checkExpressions(optimizedPlan, partitioned, "truncate");
+ checkPushedFilters(optimizedPlan, equal(truncate("data", 4), target));
+
+ List<Object[]> actual = rowsToJava(df.collectAsList());
+ Assertions.assertThat(actual.size()).isEqualTo(5);
+ }
+
+ private void checkExpressions(
+ LogicalPlan optimizedPlan, boolean partitioned, String expectedFunctionName) {
+ List<Expression> staticInvokes =
+ PlanUtils.collectSparkExpressions(
+ optimizedPlan, expression -> expression instanceof StaticInvoke);
+ Assertions.assertThat(staticInvokes).isEmpty();
+
+ List<Expression> applyExpressions =
+ PlanUtils.collectSparkExpressions(
+ optimizedPlan, expression -> expression instanceof ApplyFunctionExpression);
+
+ if (partitioned) {
+ Assertions.assertThat(applyExpressions).isEmpty();
+ } else {
+ Assertions.assertThat(applyExpressions.size()).isEqualTo(1);
+ ApplyFunctionExpression expression = (ApplyFunctionExpression) applyExpressions.get(0);
+ Assertions.assertThat(expression.name()).isEqualTo(expectedFunctionName);
+ }
+ }
+
+ private void checkPushedFilters(
+ LogicalPlan optimizedPlan, org.apache.iceberg.expressions.Expression expected) {
+ List<org.apache.iceberg.expressions.Expression> pushedFilters =
+ PlanUtils.getScanPushDownFilters(optimizedPlan);
+ Assertions.assertThat(pushedFilters.size()).isEqualTo(1);
+ org.apache.iceberg.expressions.Expression actual = pushedFilters.get(0);
+ Assertions.assertThat(ExpressionUtil.equivalent(expected, actual, STRUCT, true)).isTrue();
Review Comment:
Since this is asserting a boolean value, it would be better to have add a description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1321442714
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceStaticInvoke.scala:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.iceberg.spark.functions.SparkFunctions
+import org.apache.spark.sql.catalyst.SQLConfHelper
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.FILTER
+import org.apache.spark.sql.connector.catalog.functions.ScalarFunction
+import org.apache.spark.sql.types.StructField
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Spark analyzes the Iceberg system function to {@link StaticInvoke} which could not be pushed
+ * down to datasource. This rule will replace {@link StaticInvoke} to
+ * {@link ApplyFunctionExpression} for Iceberg system function in a filter condition.
+ */
+object ReplaceStaticInvoke extends Rule[LogicalPlan] with SQLConfHelper {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.transformWithPruning (
+ _.containsAnyPattern(FILTER)) {
+
+ case filter @ Filter(condition, _) =>
Review Comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#issuecomment-1661371315
@rdblue Yes, this is the next one. I need to rebase and add more SQL UTs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1316717252
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceStaticInvoke.scala:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.iceberg.spark.SparkSQLProperties
+import org.apache.iceberg.spark.functions.SparkFunctions
+import org.apache.iceberg.spark.source.SparkTable
+import org.apache.spark.sql.catalyst.SQLConfHelper
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.connector.catalog.Table
+import org.apache.spark.sql.connector.catalog.functions.ScalarFunction
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation
+import org.apache.spark.sql.types.StructField
+import org.apache.spark.sql.types.StructType
+import scala.annotation.tailrec
+
+/**
+ * Spark analyzes the Iceberg system function to {@link StaticInvoke} which could not be pushed
+ * down to datasource. This rule will replace {@link StaticInvoke} to
+ * {@link ApplyFunctionExpression} for Iceberg system function in a filter condition.
+ */
+object ReplaceStaticInvoke extends Rule[LogicalPlan] with SQLConfHelper {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ if (systemFuncPushDownEnabled()) {
+ rewrite(plan)
+ } else {
+ plan
+ }
+ }
+
+ private def rewrite(plan: LogicalPlan): LogicalPlan = plan transform {
+ case filter @ Filter(condition, child) if icebergRelation(child) =>
+ val newCondition = replaceStaticInvoke(condition)
+ if (newCondition fastEquals condition) {
+ filter
+ } else {
+ filter.copy(condition = newCondition)
+ }
+ }
+
+ private def systemFuncPushDownEnabled(): Boolean = {
+ conf.getConfString(
+ SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED,
+ SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED_DEFAULT.toString
+ ).toBoolean
+ }
+
+ private def replaceStaticInvoke(expression: Expression): Expression = expression.transform {
+ case invoke: StaticInvoke =>
+ invoke.functionName match {
+ case "invoke" =>
+ if (invoke.arguments.forall(_.foldable)) {
+ // The invoke should be folded into constant
+ return invoke
+ }
+
+ // Adaptive from `resolveV2Function` in org.apache.spark.sql.catalyst.analysis.ResolveFunctions
+ val unbound = SparkFunctions.loadFunctionByClass(invoke.staticObject)
+ if (unbound == null) {
+ return invoke
+ }
+
+ val inputType = StructType(invoke.arguments.zipWithIndex.map {
+ case (exp, pos) => StructField(s"_$pos", exp.dataType, exp.nullable)
+ })
+
+ val bound = try {
+ unbound.bind(inputType)
+ } catch {
+ case _: Exception =>
+ return invoke
+ }
+
+ if (bound.inputTypes().length != invoke.arguments.length) {
+ return invoke
+ }
+
+ bound match {
+ case scalarFunc: ScalarFunction[_] =>
+ ApplyFunctionExpression(scalarFunc, invoke.arguments)
+ case _ => invoke
+ }
+ }
+ }
+
+ @tailrec
+ private def icebergRelation(plan: LogicalPlan): Boolean = {
Review Comment:
Relaxed the restriction to all filters.
> Could we use something like PhysicalOperation that Spark uses to match scans, filters, and projections?
We need to get the `Filter` plan because we need to replace the filter's condition. It would be difficult to rebuild the original plan after using something like `PhysicalOperation `.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1316718056
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceStaticInvoke.scala:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.iceberg.spark.SparkSQLProperties
+import org.apache.iceberg.spark.functions.SparkFunctions
+import org.apache.iceberg.spark.source.SparkTable
+import org.apache.spark.sql.catalyst.SQLConfHelper
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.connector.catalog.Table
+import org.apache.spark.sql.connector.catalog.functions.ScalarFunction
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation
+import org.apache.spark.sql.types.StructField
+import org.apache.spark.sql.types.StructType
+import scala.annotation.tailrec
+
+/**
+ * Spark analyzes the Iceberg system function to {@link StaticInvoke} which could not be pushed
+ * down to datasource. This rule will replace {@link StaticInvoke} to
+ * {@link ApplyFunctionExpression} for Iceberg system function in a filter condition.
+ */
+object ReplaceStaticInvoke extends Rule[LogicalPlan] with SQLConfHelper {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ if (systemFuncPushDownEnabled()) {
+ rewrite(plan)
+ } else {
+ plan
+ }
+ }
+
+ private def rewrite(plan: LogicalPlan): LogicalPlan = plan transform {
+ case filter @ Filter(condition, child) if icebergRelation(child) =>
+ val newCondition = replaceStaticInvoke(condition)
+ if (newCondition fastEquals condition) {
+ filter
+ } else {
+ filter.copy(condition = newCondition)
+ }
+ }
+
+ private def systemFuncPushDownEnabled(): Boolean = {
+ conf.getConfString(
+ SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED,
+ SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED_DEFAULT.toString
+ ).toBoolean
+ }
+
+ private def replaceStaticInvoke(expression: Expression): Expression = expression.transform {
+ case invoke: StaticInvoke =>
+ invoke.functionName match {
+ case "invoke" =>
+ if (invoke.arguments.forall(_.foldable)) {
+ // The invoke should be folded into constant
+ return invoke
+ }
+
+ // Adaptive from `resolveV2Function` in org.apache.spark.sql.catalyst.analysis.ResolveFunctions
+ val unbound = SparkFunctions.loadFunctionByClass(invoke.staticObject)
+ if (unbound == null) {
+ return invoke
+ }
+
+ val inputType = StructType(invoke.arguments.zipWithIndex.map {
+ case (exp, pos) => StructField(s"_$pos", exp.dataType, exp.nullable)
+ })
+
+ val bound = try {
+ unbound.bind(inputType)
+ } catch {
+ case _: Exception =>
+ return invoke
+ }
+
+ if (bound.inputTypes().length != invoke.arguments.length) {
+ return invoke
+ }
+
+ bound match {
+ case scalarFunc: ScalarFunction[_] =>
+ ApplyFunctionExpression(scalarFunc, invoke.arguments)
+ case _ => invoke
+ }
+ }
+ }
+
+ @tailrec
+ private def icebergRelation(plan: LogicalPlan): Boolean = {
+
+ def isIcebergTable(table: Table): Boolean = table match {
+ case _: SparkTable => true
+ case _ => false
+ }
+
+ plan match {
+ case relation: DataSourceV2Relation =>
+ isIcebergTable(relation.table)
+ case scanRelation: DataSourceV2ScanRelation =>
+ icebergRelation(scanRelation.relation)
Review Comment:
Thanks for the correction. This is not needed anymore since we have relaxed the restriction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1316514218
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceStaticInvoke.scala:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.iceberg.spark.SparkSQLProperties
+import org.apache.iceberg.spark.functions.SparkFunctions
+import org.apache.iceberg.spark.source.SparkTable
+import org.apache.spark.sql.catalyst.SQLConfHelper
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.connector.catalog.Table
+import org.apache.spark.sql.connector.catalog.functions.ScalarFunction
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation
+import org.apache.spark.sql.types.StructField
+import org.apache.spark.sql.types.StructType
+import scala.annotation.tailrec
+
+/**
+ * Spark analyzes the Iceberg system function to {@link StaticInvoke} which could not be pushed
+ * down to datasource. This rule will replace {@link StaticInvoke} to
+ * {@link ApplyFunctionExpression} for Iceberg system function in a filter condition.
+ */
+object ReplaceStaticInvoke extends Rule[LogicalPlan] with SQLConfHelper {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ if (systemFuncPushDownEnabled()) {
+ rewrite(plan)
+ } else {
+ plan
+ }
+ }
+
+ private def rewrite(plan: LogicalPlan): LogicalPlan = plan transform {
+ case filter @ Filter(condition, child) if icebergRelation(child) =>
+ val newCondition = replaceStaticInvoke(condition)
+ if (newCondition fastEquals condition) {
+ filter
+ } else {
+ filter.copy(condition = newCondition)
+ }
+ }
+
+ private def systemFuncPushDownEnabled(): Boolean = {
+ conf.getConfString(
+ SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED,
+ SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED_DEFAULT.toString
+ ).toBoolean
Review Comment:
Why does this setting matter? I think that function pushdown should always be enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285567349
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestSystemFunctionPushDownDQL.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.extensions;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.createPartitionedTable;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.createUnpartitionedTable;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.days;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.hours;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.months;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.years;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.spark.SparkCatalogConfig;
+import org.apache.iceberg.spark.SparkSQLProperties;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.assertj.core.api.Assertions;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runners.Parameterized;
+
+public class TestSystemFunctionPushDownDQL extends SparkExtensionsTestBase {
+ private final String fileFormat;
+
+ public TestSystemFunctionPushDownDQL(
Review Comment:
This is tests for scan related. It would be better to include this as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285269027
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java:
##########
@@ -2728,6 +2736,105 @@ public void testMergeToWapBranchWithTableBranchIdentifier() {
branch)));
}
+ @Test
+ public void testMergeFilterWithSystemFunctionsOnPartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createPartitionedTable(spark, tableName, "years(ts)", tableProperties);
+
+ createOrReplaceView(
+ "source",
+ "id BIGINT, ts TIMESTAMP, data STRING",
+ "{ \"id\": 0, \"ts\": \"2023-08-01T09:20:44.294658+00:00\", \"data\": \"new-data-0\"}\n"
+ + "{ \"id\": 5, \"ts\": \"2023-08-02T09:23:44.294658+00:00\", \"data\": \"new-data-5\"}\n");
+
+ String date = "2017-11-22";
+ String resultSql =
+ sqlFormat("SELECT id, ts, IF(id = 0, 'new-data-0', data) FROM %s ORDER BY id", tableName);
Review Comment:
Why is there an `IF` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285267397
##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetBloomRowGroupFilter.java:
##########
@@ -334,5 +335,10 @@ private <T> boolean shouldRead(
return ROWS_MIGHT_MATCH;
}
}
+
+ @Override
+ public <T> Boolean handleNonReference(Bound<T> term) {
+ return ROWS_MIGHT_MATCH;
Review Comment:
It sounds like this is a consequence of the last PR, #7886, but I don't see how it is related to the rewrites that are the purpose of this one. Can you move these changes to a separate PR to be reviewed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285987786
##########
spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/SystemFunctionPushDownHelper.java:
##########
@@ -29,6 +31,21 @@ public static void createUnpartitionedTable(SparkSession spark, String tableName
insertRecords(spark, tableName);
}
+ public static void createUnpartitionedTable(
+ SparkSession spark, String tableName, Map<String, String> props) {
+ if (props.isEmpty()) {
+ createUnpartitionedTable(spark, tableName);
+ } else {
+ String tblProps = Joiner.on("','").withKeyValueSeparator("'='").join(props);
+ sql(
+ spark,
+ "CREATE TABLE %s (id BIGINT, ts TIMESTAMP, data STRING) USING iceberg TBLPROPERTIES ('%s')",
Review Comment:
There is a branch checking:
```java
if (props.isEmpty()) {
createUnpartitionedTable(spark, tableName);
} else {
String propsString = Joiner.on("','").withKeyValueSeparator("'='").join(props);
sql(
spark,
"CREATE TABLE %s (id BIGINT, ts TIMESTAMP, data STRING) USING iceberg TBLPROPERTIES ('%s')",
tableName,
propsString);
insertRecords(spark, tableName);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285984344
##########
spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/SparkTestBase.java:
##########
@@ -116,8 +116,12 @@ protected long waitUntilAfter(long timestampMillis) {
return current;
}
+ protected String sqlFormat(String query, Object... args) {
Review Comment:
Reverted.
##########
spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/SystemFunctionPushDownHelper.java:
##########
@@ -39,6 +56,22 @@ public static void createPartitionedTable(
insertRecords(spark, tableName);
}
+ public static void createPartitionedTable(
+ SparkSession spark, String tableName, String partitionCol, Map<String, String> props) {
+ if (props.isEmpty()) {
+ createPartitionedTable(spark, tableName, partitionCol);
+ } else {
+ String tblProps = Joiner.on("','").withKeyValueSeparator("'='").join(props);
Review Comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1290866044
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java:
##########
@@ -63,4 +63,9 @@ private SparkSQLProperties() {}
// Controls the WAP branch used for write-audit-publish workflow.
// When set, new snapshots will be committed to this branch.
public static final String WAP_BRANCH = "spark.wap.branch";
+
+ // Controls whether to push down Iceberg system function
+ public static final String SYSTEM_FUNC_PUSH_DOWN_ENABLED =
Review Comment:
Thanks @rdblue for your understanding.
> I think it may be cleaner to choose when to convert StaticInvoke rather than using a wrapper.
This is what I did in the first commit here: https://github.com/apache/iceberg/pull/8088/commits/ba7a1440b594e67cb82ad087d7a3a5b3d5aec75a#diff-66b39085c38c19ba9906ad617b851b9dcf520901a3b56976114d9a90f0fce82e. I will change back to the implementation if it is the right direction. Some things need to be mentioned in this way:
1. We need to rebuild the system function with the `StaticInvoke` https://github.com/apache/iceberg/commit/ba7a1440b594e67cb82ad087d7a3a5b3d5aec75a#diff-ee8e519e4ad1c936c1a07aacbacf71a3975f4c352fbe7810a6a99341472a128b
2. The filter condition in a normal logical plan could be easy to handle. While there is a little complicated for row-level operation because the `Filter` is not a `child plan` in the tree. We have to align the operation in the [RowLevelCommandScanRelationPushDown](https://github.com/apache/iceberg/blob/b033de7ed0a408e542013c8c9d48ec556c8e54c6/spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/RowLevelCommandScanRelationPushDown.scala#L45) to do the conversion.
> Converting in a rule would allow us to check whether the StaticInvoke expression would be folded and leave it alone.
The constant `StaticInvoke` will be folded by the optimizer.
> There could be a problem with that approach, which is that Spark might apply a rule that keeps convert back to StaticInvoke. If needed, maybe we could inject a wrapper at that point, or create a subclass of ApplyFunctionExpression that doesn't match the rule.
As of now, I have not seen the rule for it. We could take an eye out for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1321449631
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestSystemFunctionPushDownDQL.java:
##########
@@ -0,0 +1,338 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.extensions;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT;
+import static org.apache.iceberg.expressions.Expressions.bucket;
+import static org.apache.iceberg.expressions.Expressions.day;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.hour;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.month;
+import static org.apache.iceberg.expressions.Expressions.notEqual;
+import static org.apache.iceberg.expressions.Expressions.truncate;
+import static org.apache.iceberg.expressions.Expressions.year;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.STRUCT;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.createPartitionedTable;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.createUnpartitionedTable;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.timestampStrToDayOrdinal;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.timestampStrToHourOrdinal;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.timestampStrToMonthOrdinal;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.timestampStrToYearOrdinal;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.expressions.ExpressionUtil;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.spark.SparkCatalogConfig;
+import org.apache.iceberg.spark.source.PlanUtils;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression;
+import org.apache.spark.sql.catalyst.expressions.Expression;
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke;
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan;
+import org.assertj.core.api.Assertions;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runners.Parameterized;
+
+public class TestSystemFunctionPushDownDQL extends SparkExtensionsTestBase {
+ private final String fileFormat;
+
+ public TestSystemFunctionPushDownDQL(
+ String catalogName, String implementation, Map<String, String> config, String fileFormat) {
+ super(catalogName, implementation, config);
+ this.fileFormat = fileFormat;
+ }
+
+ @Parameterized.Parameters(
+ name = "catalogName = {0}, implementation = {1}, config = {2}, fileFormat = {3}")
+ public static Object[][] parameters() {
+ return new Object[][] {
+ {
+ SparkCatalogConfig.HIVE.catalogName(),
+ SparkCatalogConfig.HIVE.implementation(),
+ SparkCatalogConfig.HIVE.properties(),
+ "parquet"
Review Comment:
Thanks for the implementation.
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestSystemFunctionPushDownDQL.java:
##########
@@ -0,0 +1,338 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.extensions;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT;
+import static org.apache.iceberg.expressions.Expressions.bucket;
+import static org.apache.iceberg.expressions.Expressions.day;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.hour;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.month;
+import static org.apache.iceberg.expressions.Expressions.notEqual;
+import static org.apache.iceberg.expressions.Expressions.truncate;
+import static org.apache.iceberg.expressions.Expressions.year;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.STRUCT;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.createPartitionedTable;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.createUnpartitionedTable;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.timestampStrToDayOrdinal;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.timestampStrToHourOrdinal;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.timestampStrToMonthOrdinal;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.timestampStrToYearOrdinal;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.expressions.ExpressionUtil;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.spark.SparkCatalogConfig;
+import org.apache.iceberg.spark.source.PlanUtils;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression;
+import org.apache.spark.sql.catalyst.expressions.Expression;
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke;
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan;
+import org.assertj.core.api.Assertions;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runners.Parameterized;
+
+public class TestSystemFunctionPushDownDQL extends SparkExtensionsTestBase {
+ private final String fileFormat;
+
+ public TestSystemFunctionPushDownDQL(
+ String catalogName, String implementation, Map<String, String> config, String fileFormat) {
+ super(catalogName, implementation, config);
+ this.fileFormat = fileFormat;
+ }
+
+ @Parameterized.Parameters(
+ name = "catalogName = {0}, implementation = {1}, config = {2}, fileFormat = {3}")
+ public static Object[][] parameters() {
+ return new Object[][] {
+ {
+ SparkCatalogConfig.HIVE.catalogName(),
+ SparkCatalogConfig.HIVE.implementation(),
+ SparkCatalogConfig.HIVE.properties(),
+ "parquet"
+ },
+ {
+ SparkCatalogConfig.HIVE.catalogName(),
+ SparkCatalogConfig.HIVE.implementation(),
+ SparkCatalogConfig.HIVE.properties(),
+ "orc"
+ },
+ {
+ SparkCatalogConfig.HIVE.catalogName(),
+ SparkCatalogConfig.HIVE.implementation(),
+ SparkCatalogConfig.HIVE.properties(),
+ "avro"
+ },
+ };
+ }
+
+ @Before
+ public void before() {
+ sql("USE %s", catalogName);
+ }
+
+ @After
+ public void removeTables() {
+ sql("DROP TABLE IF EXISTS %s", tableName);
+ }
+
+ @Test
+ public void testYearsFunctionOnUnpartitionedTable() {
+ createUnpartitionedTable(spark, tableName, ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testYearsFunction(false);
+ }
+
+ @Test
+ public void testYearsFunctionOnPartitionedTable() {
+ createPartitionedTable(
+ spark, tableName, "years(ts)", ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testYearsFunction(true);
+ }
+
+ private void testYearsFunction(boolean partitioned) {
+ int targetYears = timestampStrToYearOrdinal("2017-11-22T00:00:00.000000+00:00");
+ String query =
+ String.format(
+ "SELECT * FROM %s WHERE system.years(ts) = %s ORDER BY id", tableName, targetYears);
+
+ Dataset<Row> df = spark.sql(query);
+ LogicalPlan optimizedPlan = df.queryExecution().optimizedPlan();
+
+ checkExpressions(optimizedPlan, partitioned, "years");
+ checkPushedFilters(optimizedPlan, equal(year("ts"), targetYears));
+
+ List<Object[]> actual = rowsToJava(df.collectAsList());
+ Assertions.assertThat(actual.size()).isEqualTo(5);
+ }
+
+ @Test
+ public void testMonthsFunctionOnUnpartitionedTable() {
+ createUnpartitionedTable(spark, tableName, ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testMonthsFunction(false);
+ }
+
+ @Test
+ public void testMonthsFunctionOnPartitionedTable() {
+ createPartitionedTable(
+ spark, tableName, "months(ts)", ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testMonthsFunction(true);
+ }
+
+ private void testMonthsFunction(boolean partitioned) {
+ int targetMonths = timestampStrToMonthOrdinal("2017-11-22T00:00:00.000000+00:00");
+ String query =
+ String.format(
+ "SELECT * FROM %s WHERE system.months(ts) > %s ORDER BY id", tableName, targetMonths);
+
+ Dataset<Row> df = spark.sql(query);
+ LogicalPlan optimizedPlan = df.queryExecution().optimizedPlan();
+
+ checkExpressions(optimizedPlan, partitioned, "months");
+ checkPushedFilters(optimizedPlan, greaterThan(month("ts"), targetMonths));
+
+ List<Object[]> actual = rowsToJava(df.collectAsList());
+ Assertions.assertThat(actual.size()).isEqualTo(5);
+ }
+
+ @Test
+ public void testDaysFunctionOnUnpartitionedTable() {
+ createUnpartitionedTable(spark, tableName, ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testDaysFunction(false);
+ }
+
+ @Test
+ public void testDaysFunctionOnPartitionedTable() {
+ createPartitionedTable(
+ spark, tableName, "days(ts)", ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testDaysFunction(true);
+ }
+
+ private void testDaysFunction(boolean partitioned) {
+ String timestamp = "2018-11-20T00:00:00.000000+00:00";
+ int targetDays = timestampStrToDayOrdinal(timestamp);
+ String query =
+ String.format(
+ "SELECT * FROM %s WHERE system.days(ts) < date('%s') ORDER BY id",
+ tableName, timestamp);
+
+ Dataset<Row> df = spark.sql(query);
+ LogicalPlan optimizedPlan = df.queryExecution().optimizedPlan();
+
+ checkExpressions(optimizedPlan, partitioned, "days");
+ checkPushedFilters(optimizedPlan, lessThan(day("ts"), targetDays));
+
+ List<Object[]> actual = rowsToJava(df.collectAsList());
+ Assertions.assertThat(actual.size()).isEqualTo(5);
+ }
+
+ @Test
+ public void testHoursFunctionOnUnpartitionedTable() {
+ createUnpartitionedTable(spark, tableName, ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testHoursFunction(false);
+ }
+
+ @Test
+ public void testHoursFunctionOnPartitionedTable() {
+ createPartitionedTable(
+ spark, tableName, "hours(ts)", ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testHoursFunction(true);
+ }
+
+ private void testHoursFunction(boolean partitioned) {
+ int targetHours = timestampStrToHourOrdinal("2017-11-22T06:02:09.243857+00:00");
+ String query =
+ String.format(
+ "SELECT * FROM %s WHERE system.hours(ts) >= %s ORDER BY id", tableName, targetHours);
+
+ Dataset<Row> df = spark.sql(query);
+ LogicalPlan optimizedPlan = df.queryExecution().optimizedPlan();
+
+ checkExpressions(optimizedPlan, partitioned, "hours");
+ checkPushedFilters(optimizedPlan, greaterThanOrEqual(hour("ts"), targetHours));
+
+ List<Object[]> actual = rowsToJava(df.collectAsList());
+ Assertions.assertThat(actual.size()).isEqualTo(8);
+ }
+
+ @Test
+ public void testBucketLongFunctionOnUnpartitionedTable() {
+ createUnpartitionedTable(spark, tableName, ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testBucketLongFunction(false);
+ }
+
+ @Test
+ public void testBucketLongFunctionOnPartitionedTable() {
+ createPartitionedTable(
+ spark, tableName, "bucket(5, id)", ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testBucketLongFunction(true);
+ }
+
+ private void testBucketLongFunction(boolean partitioned) {
+ int target = 2;
+ String query =
+ String.format(
+ "SELECT * FROM %s WHERE system.bucket(5, id) <= %s ORDER BY id", tableName, target);
+
+ Dataset<Row> df = spark.sql(query);
+ LogicalPlan optimizedPlan = df.queryExecution().optimizedPlan();
+
+ checkExpressions(optimizedPlan, partitioned, "bucket");
+ checkPushedFilters(optimizedPlan, lessThanOrEqual(bucket("id", 5), target));
+
+ List<Object[]> actual = rowsToJava(df.collectAsList());
+ Assertions.assertThat(actual.size()).isEqualTo(5);
+ }
+
+ @Test
+ public void testBucketStringFunctionOnUnpartitionedTable() {
+ createUnpartitionedTable(spark, tableName, ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testBucketStringFunction(false);
+ }
+
+ @Test
+ public void testBucketStringFunctionOnPartitionedTable() {
+ createPartitionedTable(
+ spark, tableName, "bucket(5, data)", ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testBucketStringFunction(true);
+ }
+
+ private void testBucketStringFunction(boolean partitioned) {
+ int target = 2;
+ String query =
+ String.format(
+ "SELECT * FROM %s WHERE system.bucket(5, data) != %s ORDER BY id", tableName, target);
+
+ Dataset<Row> df = spark.sql(query);
+ LogicalPlan optimizedPlan = df.queryExecution().optimizedPlan();
+
+ checkExpressions(optimizedPlan, partitioned, "bucket");
+ checkPushedFilters(optimizedPlan, notEqual(bucket("data", 5), target));
+
+ List<Object[]> actual = rowsToJava(df.collectAsList());
+ Assertions.assertThat(actual.size()).isEqualTo(8);
+ }
+
+ @Test
+ public void testTruncateFunctionOnUnpartitionedTable() {
+ createUnpartitionedTable(spark, tableName, ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testTruncateFunction(false);
+ }
+
+ @Test
+ public void testTruncateFunctionOnPartitionedTable() {
+ createPartitionedTable(
+ spark, tableName, "truncate(4, data)", ImmutableMap.of(DEFAULT_FILE_FORMAT, fileFormat));
+ testTruncateFunction(true);
+ }
+
+ private void testTruncateFunction(boolean partitioned) {
+ String target = "data";
+ String query =
+ String.format(
+ "SELECT * FROM %s WHERE system.truncate(4, data) = '%s' ORDER BY id",
+ tableName, target);
+
+ Dataset<Row> df = spark.sql(query);
+ LogicalPlan optimizedPlan = df.queryExecution().optimizedPlan();
+
+ checkExpressions(optimizedPlan, partitioned, "truncate");
+ checkPushedFilters(optimizedPlan, equal(truncate("data", 4), target));
+
+ List<Object[]> actual = rowsToJava(df.collectAsList());
+ Assertions.assertThat(actual.size()).isEqualTo(5);
+ }
+
+ private void checkExpressions(
+ LogicalPlan optimizedPlan, boolean partitioned, String expectedFunctionName) {
+ List<Expression> staticInvokes =
+ PlanUtils.collectSparkExpressions(
+ optimizedPlan, expression -> expression instanceof StaticInvoke);
+ Assertions.assertThat(staticInvokes).isEmpty();
+
+ List<Expression> applyExpressions =
+ PlanUtils.collectSparkExpressions(
+ optimizedPlan, expression -> expression instanceof ApplyFunctionExpression);
+
+ if (partitioned) {
+ Assertions.assertThat(applyExpressions).isEmpty();
+ } else {
+ Assertions.assertThat(applyExpressions.size()).isEqualTo(1);
+ ApplyFunctionExpression expression = (ApplyFunctionExpression) applyExpressions.get(0);
+ Assertions.assertThat(expression.name()).isEqualTo(expectedFunctionName);
+ }
+ }
+
+ private void checkPushedFilters(
+ LogicalPlan optimizedPlan, org.apache.iceberg.expressions.Expression expected) {
+ List<org.apache.iceberg.expressions.Expression> pushedFilters =
+ PlanUtils.getScanPushDownFilters(optimizedPlan);
+ Assertions.assertThat(pushedFilters.size()).isEqualTo(1);
+ org.apache.iceberg.expressions.Expression actual = pushedFilters.get(0);
+ Assertions.assertThat(ExpressionUtil.equivalent(expected, actual, STRUCT, true)).isTrue();
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1316515551
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceStaticInvoke.scala:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.iceberg.spark.SparkSQLProperties
+import org.apache.iceberg.spark.functions.SparkFunctions
+import org.apache.iceberg.spark.source.SparkTable
+import org.apache.spark.sql.catalyst.SQLConfHelper
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.connector.catalog.Table
+import org.apache.spark.sql.connector.catalog.functions.ScalarFunction
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation
+import org.apache.spark.sql.types.StructField
+import org.apache.spark.sql.types.StructType
+import scala.annotation.tailrec
+
+/**
+ * Spark analyzes the Iceberg system function to {@link StaticInvoke} which could not be pushed
+ * down to datasource. This rule will replace {@link StaticInvoke} to
+ * {@link ApplyFunctionExpression} for Iceberg system function in a filter condition.
+ */
+object ReplaceStaticInvoke extends Rule[LogicalPlan] with SQLConfHelper {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ if (systemFuncPushDownEnabled()) {
+ rewrite(plan)
+ } else {
+ plan
+ }
+ }
+
+ private def rewrite(plan: LogicalPlan): LogicalPlan = plan transform {
+ case filter @ Filter(condition, child) if icebergRelation(child) =>
+ val newCondition = replaceStaticInvoke(condition)
+ if (newCondition fastEquals condition) {
+ filter
+ } else {
+ filter.copy(condition = newCondition)
+ }
+ }
+
+ private def systemFuncPushDownEnabled(): Boolean = {
+ conf.getConfString(
+ SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED,
+ SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED_DEFAULT.toString
+ ).toBoolean
+ }
+
+ private def replaceStaticInvoke(expression: Expression): Expression = expression.transform {
+ case invoke: StaticInvoke =>
+ invoke.functionName match {
+ case "invoke" =>
+ if (invoke.arguments.forall(_.foldable)) {
+ // The invoke should be folded into constant
+ return invoke
+ }
+
+ // Adaptive from `resolveV2Function` in org.apache.spark.sql.catalyst.analysis.ResolveFunctions
+ val unbound = SparkFunctions.loadFunctionByClass(invoke.staticObject)
+ if (unbound == null) {
+ return invoke
+ }
+
+ val inputType = StructType(invoke.arguments.zipWithIndex.map {
+ case (exp, pos) => StructField(s"_$pos", exp.dataType, exp.nullable)
+ })
+
+ val bound = try {
+ unbound.bind(inputType)
+ } catch {
+ case _: Exception =>
+ return invoke
+ }
+
+ if (bound.inputTypes().length != invoke.arguments.length) {
+ return invoke
+ }
+
+ bound match {
+ case scalarFunc: ScalarFunction[_] =>
+ ApplyFunctionExpression(scalarFunc, invoke.arguments)
+ case _ => invoke
+ }
+ }
+ }
+
+ @tailrec
+ private def icebergRelation(plan: LogicalPlan): Boolean = {
Review Comment:
Why not do this everywhere? I'm concerned that we may not have a case where a filter is directly on top of the v2 relation. Could we use something like `PhysicalOperation` that Spark uses to match scans, filters, and projections?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1316714452
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceStaticInvoke.scala:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.iceberg.spark.SparkSQLProperties
+import org.apache.iceberg.spark.functions.SparkFunctions
+import org.apache.iceberg.spark.source.SparkTable
+import org.apache.spark.sql.catalyst.SQLConfHelper
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.connector.catalog.Table
+import org.apache.spark.sql.connector.catalog.functions.ScalarFunction
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation
+import org.apache.spark.sql.types.StructField
+import org.apache.spark.sql.types.StructType
+import scala.annotation.tailrec
+
+/**
+ * Spark analyzes the Iceberg system function to {@link StaticInvoke} which could not be pushed
+ * down to datasource. This rule will replace {@link StaticInvoke} to
+ * {@link ApplyFunctionExpression} for Iceberg system function in a filter condition.
+ */
+object ReplaceStaticInvoke extends Rule[LogicalPlan] with SQLConfHelper {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ if (systemFuncPushDownEnabled()) {
+ rewrite(plan)
+ } else {
+ plan
+ }
+ }
+
+ private def rewrite(plan: LogicalPlan): LogicalPlan = plan transform {
+ case filter @ Filter(condition, child) if icebergRelation(child) =>
+ val newCondition = replaceStaticInvoke(condition)
+ if (newCondition fastEquals condition) {
+ filter
+ } else {
+ filter.copy(condition = newCondition)
+ }
+ }
+
+ private def systemFuncPushDownEnabled(): Boolean = {
+ conf.getConfString(
+ SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED,
+ SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED_DEFAULT.toString
+ ).toBoolean
Review Comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1282931140
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java:
##########
@@ -63,4 +63,9 @@ private SparkSQLProperties() {}
// Controls the WAP branch used for write-audit-publish workflow.
// When set, new snapshots will be committed to this branch.
public static final String WAP_BRANCH = "spark.wap.branch";
+
+ // Controls whether to push down Iceberg system function
+ public static final String SYSTEM_FUNC_PUSH_DOWN_ENABLED =
Review Comment:
As opposed to `StaticInvoke`, `ApplyFunctionExpression` can not code gen. So add a parameter to control enable the converting or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#issuecomment-1663687889
@rdblue, just lacks tests for MergeInto, and will add them later. But I think it is ready for review the other parts.
Also, cc @RussellSpitzer @szehon-ho @huaxingao @holdenk who have helped review #7886. Thanks in advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285216508
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteStaticInvoke.scala:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.iceberg.spark.SparkSQLProperties
+import org.apache.iceberg.spark.functions.SparkFunctions
+import org.apache.iceberg.spark.source.SparkTable
+import org.apache.spark.sql.catalyst.SQLConfHelper
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke
+import org.apache.spark.sql.catalyst.planning.RewrittenRowLevelCommand
+import org.apache.spark.sql.catalyst.planning.ScanOperation
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.NoStatsUnaryNode
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceIcebergData
+import org.apache.spark.sql.catalyst.plans.logical.RowLevelCommand
+import org.apache.spark.sql.catalyst.plans.logical.WriteDelta
+import org.apache.spark.sql.catalyst.plans.logical.WriteIcebergDelta
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.connector.catalog.Table
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation
+import org.apache.spark.sql.execution.datasources.v2.UnplannedGroupBasedMergeOperation
+import org.apache.spark.sql.types.AbstractDataType
+import org.apache.spark.sql.types.DataType
+import scala.annotation.tailrec
+
+/**
+ * Spark analyzes the Iceberg system function to {@link StaticInvoke} which could not be pushed
+ * down to datasource. This rule will replace {@link StaticInvoke} to
+ * {@link ApplyFunctionExpression} for Iceberg system function in a filter condition.
+ */
+case object RewriteStaticInvoke extends Rule[LogicalPlan] with SQLConfHelper {
Review Comment:
Changed the implementation to the second choice to avoid rewriting the plan in runtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285268304
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java:
##########
@@ -1304,6 +1313,82 @@ public void testDeleteToCustomWapBranchWithoutWhereClause() throws NoSuchTableEx
});
}
+ @Test
+ public void testDeleteFilterWithSystemFuncOnPartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createPartitionedTable(spark, tableName, "years(ts)", tableProperties);
+
+ String date = "2018-12-21";
+ String remainSql =
+ sqlFormat(
+ "SELECT * FROM %s WHERE %s.system.years(ts) != %s.system.years(date('%s')) ORDER BY id",
+ tableName, catalogName, catalogName, date);
+
+ List<Object[]> expected = sql(remainSql);
+
+ withSQLConf(
+ ImmutableMap.of(SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED, "true"),
+ () -> {
+ String deleteSql =
+ sqlFormat(
+ "DELETE FROM %s WHERE %s.system.years(ts) == %s.system.years(date('%s'))",
+ tableName, catalogName, catalogName, date);
+ Object explain = scalarSql("EXPLAIN EXTENDED %s", deleteSql);
+ // Metadata delete
+ Assertions.assertThat((String) explain).contains("DeleteFromTableWithFilters");
+ Assertions.assertThat((String) explain).contains("[years(ts) = " + years(date));
+
+ sql(deleteSql);
+
+ List<Object[]> actual = sql(remainSql);
+ assertEquals("Results should match", expected, actual);
+ });
+ }
+
+ @Test
+ public void testDeleteFilterWithSystemFuncOnUnpartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createUnpartitionedTable(spark, tableName, tableProperties);
+
+ String date = "2018-12-21";
+ String remainSql =
+ sqlFormat(
+ "SELECT * FROM %s WHERE %s.system.months(ts) = %s.system.months(date('%s')) ORDER BY id",
+ tableName, catalogName, catalogName, date);
+
+ List<Object[]> expected = sql(remainSql);
+
+ withSQLConf(
+ ImmutableMap.of(SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED, "true"),
+ () -> {
+ String deleteSql =
+ sqlFormat(
+ "DELETE FROM %s WHERE %s.system.months(ts) != %s.system.months(date('%s'))",
+ tableName, catalogName, catalogName, date);
+ Object explain = scalarSql("EXPLAIN EXTENDED %s", deleteSql);
+ Assertions.assertThat((String) explain)
+ .contains("Filter NOT (applyfunctionexpression(Wrapper(iceberg.months(timestamp))");
+ Assertions.assertThat((String) explain).contains("[filters=NOT (ts = " + months(date));
Review Comment:
I think this is incorrect. Why is this `ts = ` instead of `months(ts) = `?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285268601
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java:
##########
@@ -1304,6 +1313,82 @@ public void testDeleteToCustomWapBranchWithoutWhereClause() throws NoSuchTableEx
});
}
+ @Test
+ public void testDeleteFilterWithSystemFuncOnPartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createPartitionedTable(spark, tableName, "years(ts)", tableProperties);
+
+ String date = "2018-12-21";
+ String remainSql =
+ sqlFormat(
+ "SELECT * FROM %s WHERE %s.system.years(ts) != %s.system.years(date('%s')) ORDER BY id",
Review Comment:
This should not rely on constant folding (or something that might use constant folding) to produce the expected rows. If it does, then problems could affect both. Instead, this should calculate the value directly and embed it in the SQL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285270510
##########
spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/SparkTestBase.java:
##########
@@ -116,8 +116,12 @@ protected long waitUntilAfter(long timestampMillis) {
return current;
}
+ protected String sqlFormat(String query, Object... args) {
Review Comment:
Why is this needed? Formatting SQL is done by the `sql` method. Even if it is reused, that's the right place for it. Can you roll back this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285562477
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/FoldConstantSystemFunction.scala:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.iceberg.spark.functions.UnboundFunctionWrapper.SystemFunctionWrapper
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression
+import org.apache.spark.sql.catalyst.expressions.EmptyRow
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.plans.logical.LocalRelation
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * Folding those system function that applies a constant value.
+ */
+object FoldConstantSystemFunction extends Rule[LogicalPlan] {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+ case logicalPlan: LogicalPlan =>
+ logicalPlan transformExpressions {
+ case applyFunc: ApplyFunctionExpression
+ if applyFunc.function.isInstanceOf[SystemFunctionWrapper[_]] =>
+ val fakeProjectList = applyFunc.children.zipWithIndex.map(pair => Alias(pair._1, s"fake_${pair._2}")())
+ val fakePlan = Project(fakeProjectList, LocalRelation(Nil))
+ val foldedExprs =
+ ConstantFolding.apply(fakePlan).asInstanceOf[Project].projectList.map(_.asInstanceOf[Alias].child)
Review Comment:
> Why is this fake plan needed?
`ConstantFolding` takes a logic plan as input.
> Wouldn't Spark already call ConstantFolding on both ApplyFunctionExpression and StaticInvoke?
Yeah, `ApplyFunctionExpression` can not be folded because it is not `foldable`.
> so I'm not sure why StaticInvoke is not already folded.
`StaticInvoke` can be folded. Here the rule is just for `ApplyFunctionExpression` to address it unfoldable problem.
> I think we could fix that by adding this upstream:
Yeah, I think we should patch to spark. I will submit a PR for it. Here is a temporary fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jerqi commented on pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#issuecomment-1691470311
@rdblue
Dear Blue,
Could you give further reviews when you have time? I know you are very busy. I'm sorry to bother you and really appreciate your contribution to the open source community. Thanks for your careful review to make Iceberg great. However, this feature is really important. There are many users who are eager to use this feature. Your reviews will make the feature better and let more users use this feature sooner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1320853955
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceStaticInvoke.scala:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.iceberg.spark.functions.SparkFunctions
+import org.apache.spark.sql.catalyst.SQLConfHelper
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.FILTER
+import org.apache.spark.sql.connector.catalog.functions.ScalarFunction
+import org.apache.spark.sql.types.StructField
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Spark analyzes the Iceberg system function to {@link StaticInvoke} which could not be pushed
+ * down to datasource. This rule will replace {@link StaticInvoke} to
+ * {@link ApplyFunctionExpression} for Iceberg system function in a filter condition.
+ */
+object ReplaceStaticInvoke extends Rule[LogicalPlan] with SQLConfHelper {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.transformWithPruning (
+ _.containsAnyPattern(FILTER)) {
+
+ case filter @ Filter(condition, _) =>
+ val newCondition = replaceStaticInvoke(condition)
+ if (newCondition fastEquals condition) {
+ filter
+ } else {
+ filter.copy(condition = newCondition)
+ }
+ }
+
+ private def replaceStaticInvoke(expression: Expression): Expression = expression.transform {
+ case invoke: StaticInvoke =>
+ invoke.functionName match {
+ case "invoke" =>
Review Comment:
This is a constant: `ScalarFunction.MAGIC_METHOD_NAME`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1321443035
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceStaticInvoke.scala:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.iceberg.spark.functions.SparkFunctions
+import org.apache.spark.sql.catalyst.SQLConfHelper
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.FILTER
+import org.apache.spark.sql.connector.catalog.functions.ScalarFunction
+import org.apache.spark.sql.types.StructField
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Spark analyzes the Iceberg system function to {@link StaticInvoke} which could not be pushed
+ * down to datasource. This rule will replace {@link StaticInvoke} to
+ * {@link ApplyFunctionExpression} for Iceberg system function in a filter condition.
+ */
+object ReplaceStaticInvoke extends Rule[LogicalPlan] with SQLConfHelper {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.transformWithPruning (
+ _.containsAnyPattern(FILTER)) {
+
+ case filter @ Filter(condition, _) =>
+ val newCondition = replaceStaticInvoke(condition)
+ if (newCondition fastEquals condition) {
+ filter
+ } else {
+ filter.copy(condition = newCondition)
+ }
+ }
+
+ private def replaceStaticInvoke(expression: Expression): Expression = expression.transform {
+ case invoke: StaticInvoke =>
Review Comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1283917244
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteStaticInvoke.scala:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.iceberg.spark.SparkSQLProperties
+import org.apache.iceberg.spark.functions.SparkFunctions
+import org.apache.iceberg.spark.source.SparkTable
+import org.apache.spark.sql.catalyst.SQLConfHelper
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke
+import org.apache.spark.sql.catalyst.planning.RewrittenRowLevelCommand
+import org.apache.spark.sql.catalyst.planning.ScanOperation
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.NoStatsUnaryNode
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceIcebergData
+import org.apache.spark.sql.catalyst.plans.logical.RowLevelCommand
+import org.apache.spark.sql.catalyst.plans.logical.WriteDelta
+import org.apache.spark.sql.catalyst.plans.logical.WriteIcebergDelta
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.connector.catalog.Table
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation
+import org.apache.spark.sql.execution.datasources.v2.UnplannedGroupBasedMergeOperation
+import org.apache.spark.sql.types.AbstractDataType
+import org.apache.spark.sql.types.DataType
+import scala.annotation.tailrec
+
+/**
+ * Spark analyzes the Iceberg system function to {@link StaticInvoke} which could not be pushed
+ * down to datasource. This rule will replace {@link StaticInvoke} to
+ * {@link ApplyFunctionExpression} for Iceberg system function in a filter condition.
+ */
+case object RewriteStaticInvoke extends Rule[LogicalPlan] with SQLConfHelper {
Review Comment:
system function can be analyzed into `StaticInvoke` or `ApplyFunctionExpression`, while `StaticInvoke` takes higher priority. The details can be found here:
https://github.com/apache/spark/blob/4ed59d932631ba819d6b6071ee91408622a312db/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/V2ExpressionUtils.scala#L152
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285551374
##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetBloomRowGroupFilter.java:
##########
@@ -334,5 +335,10 @@ private <T> boolean shouldRead(
return ROWS_MIGHT_MATCH;
}
}
+
+ @Override
+ public <T> Boolean handleNonReference(Bound<T> term) {
+ return ROWS_MIGHT_MATCH;
Review Comment:
Submit #8243 for parquet and #8244 for orc. Will rebase after they are merged. Keep here for CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285983629
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java:
##########
@@ -2728,6 +2736,105 @@ public void testMergeToWapBranchWithTableBranchIdentifier() {
branch)));
}
+ @Test
+ public void testMergeFilterWithSystemFunctionsOnPartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createPartitionedTable(spark, tableName, "years(ts)", tableProperties);
+
+ createOrReplaceView(
+ "source",
+ "id BIGINT, ts TIMESTAMP, data STRING",
+ "{ \"id\": 0, \"ts\": \"2023-08-01T09:20:44.294658+00:00\", \"data\": \"new-data-0\"}\n"
+ + "{ \"id\": 5, \"ts\": \"2023-08-02T09:23:44.294658+00:00\", \"data\": \"new-data-5\"}\n");
+
+ String date = "2017-11-22";
+ String resultSql =
+ sqlFormat("SELECT id, ts, IF(id = 0, 'new-data-0', data) FROM %s ORDER BY id", tableName);
+ List<Object[]> expected = sql(resultSql);
+
+ withSQLConf(
+ ImmutableMap.of(SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED, "true"),
+ () -> {
+ String mergeSql =
+ sqlFormat(
+ "MERGE INTO %s AS t USING source AS s "
+ + "ON t.id = s.id AND %s.system.years(t.ts) = %s.system.years(date('%s')) "
+ + "WHEN MATCHED "
+ + " THEN UPDATE SET t.data = s.data "
+ + "WHEN NOT MATCHED AND s.id != 5 "
+ + " THEN INSERT *",
+ tableName, catalogName, catalogName, date);
+ Dataset<Row> df = spark.sql(mergeSql);
+ // Should not contain applyfunction since filter function could be evaluated completely on
+ // Iceberg side and will not return post scan filters
+ Assertions.assertThat(df.queryExecution().sparkPlan().toString())
+ .doesNotContain("Filter (applyfunctionexpression(Wrapper(iceberg.years(timestamp))");
+ Assertions.assertThat(df.queryExecution().sparkPlan().toString())
+ .contains("[filters=ts = " + years(date));
Review Comment:
The actual filter is `year(ts)` while the transform function message is lost in the string represented. Will change the checking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1286438327
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java:
##########
@@ -1304,6 +1313,82 @@ public void testDeleteToCustomWapBranchWithoutWhereClause() throws NoSuchTableEx
});
}
+ @Test
+ public void testDeleteFilterWithSystemFuncOnPartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createPartitionedTable(spark, tableName, "years(ts)", tableProperties);
+
+ String date = "2018-12-21";
+ String remainSql =
+ sqlFormat(
+ "SELECT * FROM %s WHERE %s.system.years(ts) != %s.system.years(date('%s')) ORDER BY id",
+ tableName, catalogName, catalogName, date);
+
+ List<Object[]> expected = sql(remainSql);
+
+ withSQLConf(
+ ImmutableMap.of(SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED, "true"),
+ () -> {
+ String deleteSql =
+ sqlFormat(
+ "DELETE FROM %s WHERE %s.system.years(ts) == %s.system.years(date('%s'))",
+ tableName, catalogName, catalogName, date);
+ Object explain = scalarSql("EXPLAIN EXTENDED %s", deleteSql);
+ // Metadata delete
+ Assertions.assertThat((String) explain).contains("DeleteFromTableWithFilters");
+ Assertions.assertThat((String) explain).contains("[years(ts) = " + years(date));
+
+ sql(deleteSql);
+
+ List<Object[]> actual = sql(remainSql);
+ assertEquals("Results should match", expected, actual);
+ });
+ }
+
+ @Test
+ public void testDeleteFilterWithSystemFuncOnUnpartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createUnpartitionedTable(spark, tableName, tableProperties);
+
+ String date = "2018-12-21";
+ String remainSql =
+ sqlFormat(
+ "SELECT * FROM %s WHERE %s.system.months(ts) = %s.system.months(date('%s')) ORDER BY id",
+ tableName, catalogName, catalogName, date);
+
+ List<Object[]> expected = sql(remainSql);
+
+ withSQLConf(
+ ImmutableMap.of(SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED, "true"),
+ () -> {
+ String deleteSql =
+ sqlFormat(
+ "DELETE FROM %s WHERE %s.system.months(ts) != %s.system.months(date('%s'))",
+ tableName, catalogName, catalogName, date);
+ Object explain = scalarSql("EXPLAIN EXTENDED %s", deleteSql);
+ Assertions.assertThat((String) explain)
+ .contains("Filter NOT (applyfunctionexpression(Wrapper(iceberg.months(timestamp))");
+ Assertions.assertThat((String) explain).contains("[filters=NOT (ts = " + months(date));
Review Comment:
Can you point to where it is lost? Is it a bug in Spark to fix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#issuecomment-1660644284
@ConeyLiu, is this one next after #7886?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285555970
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java:
##########
@@ -1304,6 +1313,82 @@ public void testDeleteToCustomWapBranchWithoutWhereClause() throws NoSuchTableEx
});
}
+ @Test
+ public void testDeleteFilterWithSystemFuncOnPartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createPartitionedTable(spark, tableName, "years(ts)", tableProperties);
+
+ String date = "2018-12-21";
+ String remainSql =
+ sqlFormat(
+ "SELECT * FROM %s WHERE %s.system.years(ts) != %s.system.years(date('%s')) ORDER BY id",
+ tableName, catalogName, catalogName, date);
+
+ List<Object[]> expected = sql(remainSql);
+
+ withSQLConf(
+ ImmutableMap.of(SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED, "true"),
+ () -> {
+ String deleteSql =
+ sqlFormat(
+ "DELETE FROM %s WHERE %s.system.years(ts) == %s.system.years(date('%s'))",
+ tableName, catalogName, catalogName, date);
+ Object explain = scalarSql("EXPLAIN EXTENDED %s", deleteSql);
+ // Metadata delete
+ Assertions.assertThat((String) explain).contains("DeleteFromTableWithFilters");
+ Assertions.assertThat((String) explain).contains("[years(ts) = " + years(date));
+
+ sql(deleteSql);
+
+ List<Object[]> actual = sql(remainSql);
+ assertEquals("Results should match", expected, actual);
+ });
+ }
+
+ @Test
+ public void testDeleteFilterWithSystemFuncOnUnpartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createUnpartitionedTable(spark, tableName, tableProperties);
+
+ String date = "2018-12-21";
+ String remainSql =
+ sqlFormat(
+ "SELECT * FROM %s WHERE %s.system.months(ts) = %s.system.months(date('%s')) ORDER BY id",
+ tableName, catalogName, catalogName, date);
+
+ List<Object[]> expected = sql(remainSql);
+
+ withSQLConf(
+ ImmutableMap.of(SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED, "true"),
+ () -> {
+ String deleteSql =
+ sqlFormat(
+ "DELETE FROM %s WHERE %s.system.months(ts) != %s.system.months(date('%s'))",
+ tableName, catalogName, catalogName, date);
+ Object explain = scalarSql("EXPLAIN EXTENDED %s", deleteSql);
+ Assertions.assertThat((String) explain)
+ .contains("Filter NOT (applyfunctionexpression(Wrapper(iceberg.months(timestamp))");
+ Assertions.assertThat((String) explain).contains("[filters=NOT (ts = " + months(date));
Review Comment:
The actual filter is `month(ts)` while the transform function message is lost in the string represented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285551672
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/RowLevelCommandScanRelationPushDown.scala:
##########
@@ -122,16 +130,15 @@ object RowLevelCommandScanRelationPushDown extends Rule[LogicalPlan] with Predic
private def pushFilters(
cond: Expression,
scanBuilder: ScanBuilder,
- tableAttrs: Seq[AttributeReference]): (Seq[Filter], Seq[Predicate]) = {
+ tableAttrs: Seq[AttributeReference]): (Either[Seq[Filter], Seq[Predicate]], Seq[Expression]) = {
val tableAttrSet = AttributeSet(tableAttrs)
val filters = splitConjunctivePredicates(cond).filter(_.references.subsetOf(tableAttrSet))
val normalizedFilters = DataSourceStrategy.normalizeExprs(filters, tableAttrs)
val (_, normalizedFiltersWithoutSubquery) =
normalizedFilters.partition(SubqueryExpression.hasSubquery)
- val (pushedFilters, _) = PushDownUtils.pushFilters(scanBuilder, normalizedFiltersWithoutSubquery)
- (pushedFilters.left.getOrElse(Seq.empty), pushedFilters.right.getOrElse(Seq.empty))
+ PushDownUtils.pushFilters(scanBuilder, normalizedFiltersWithoutSubquery)
Review Comment:
Submit #8249
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1286556424
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java:
##########
@@ -1304,6 +1313,82 @@ public void testDeleteToCustomWapBranchWithoutWhereClause() throws NoSuchTableEx
});
}
+ @Test
+ public void testDeleteFilterWithSystemFuncOnPartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createPartitionedTable(spark, tableName, "years(ts)", tableProperties);
+
+ String date = "2018-12-21";
+ String remainSql =
+ sqlFormat(
+ "SELECT * FROM %s WHERE %s.system.years(ts) != %s.system.years(date('%s')) ORDER BY id",
+ tableName, catalogName, catalogName, date);
+
+ List<Object[]> expected = sql(remainSql);
+
+ withSQLConf(
+ ImmutableMap.of(SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED, "true"),
+ () -> {
+ String deleteSql =
+ sqlFormat(
+ "DELETE FROM %s WHERE %s.system.years(ts) == %s.system.years(date('%s'))",
+ tableName, catalogName, catalogName, date);
+ Object explain = scalarSql("EXPLAIN EXTENDED %s", deleteSql);
+ // Metadata delete
+ Assertions.assertThat((String) explain).contains("DeleteFromTableWithFilters");
+ Assertions.assertThat((String) explain).contains("[years(ts) = " + years(date));
+
+ sql(deleteSql);
+
+ List<Object[]> actual = sql(remainSql);
+ assertEquals("Results should match", expected, actual);
+ });
+ }
+
+ @Test
+ public void testDeleteFilterWithSystemFuncOnUnpartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createUnpartitionedTable(spark, tableName, tableProperties);
+
+ String date = "2018-12-21";
+ String remainSql =
+ sqlFormat(
+ "SELECT * FROM %s WHERE %s.system.months(ts) = %s.system.months(date('%s')) ORDER BY id",
+ tableName, catalogName, catalogName, date);
+
+ List<Object[]> expected = sql(remainSql);
+
+ withSQLConf(
+ ImmutableMap.of(SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED, "true"),
+ () -> {
+ String deleteSql =
+ sqlFormat(
+ "DELETE FROM %s WHERE %s.system.months(ts) != %s.system.months(date('%s'))",
+ tableName, catalogName, catalogName, date);
+ Object explain = scalarSql("EXPLAIN EXTENDED %s", deleteSql);
+ Assertions.assertThat((String) explain)
+ .contains("Filter NOT (applyfunctionexpression(Wrapper(iceberg.months(timestamp))");
+ Assertions.assertThat((String) explain).contains("[filters=NOT (ts = " + months(date));
Review Comment:
I am deep into it. It should be caused by Iceberg side. The related code at here: https://github.com/apache/iceberg/blob/00f771de3a71c4de39e310114f0f776c7a81838d/spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/Spark3Util.java#L589. I will try to fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285270354
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/functions/SparkFunctions.java:
##########
@@ -52,6 +55,24 @@ public static List<String> list() {
public static UnboundFunction load(String name) {
// function resolution is case-insensitive to match the existing Spark behavior for functions
- return FUNCTIONS.get(name.toLowerCase(Locale.ROOT));
+ String lowerCaseName = name.toLowerCase(Locale.ROOT);
+ UnboundFunction function = FUNCTIONS.get(lowerCaseName);
+
+ SparkSession sparkSession = SparkSession.active();
+ String enabledStr =
+ sparkSession
+ .conf()
+ .get(
+ SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED,
+ String.valueOf(SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED_DEFAULT));
+ if (Boolean.parseBoolean(enabledStr)
+ && SparkV2Filters.SUPPORTED_FUNCTIONS.contains(lowerCaseName)) {
+ // Use a UnboundFunctionWrapper to wrap the actual system function. The system function will
+ // be
+ // analyzed into ApplyFunctionExpression which could be pushed down datasource.
+ return new UnboundFunctionWrapper(function);
Review Comment:
As I said above, having a boolean that controls whether we wrap to trick Spark into not using `StaticInvoke` is a problem. I think that we probably need to support constant folding with `StaticInvoke`, and should probably also fix this on the Spark side. I don't see why we need to do this in Iceberg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285269027
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java:
##########
@@ -2728,6 +2736,105 @@ public void testMergeToWapBranchWithTableBranchIdentifier() {
branch)));
}
+ @Test
+ public void testMergeFilterWithSystemFunctionsOnPartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createPartitionedTable(spark, tableName, "years(ts)", tableProperties);
+
+ createOrReplaceView(
+ "source",
+ "id BIGINT, ts TIMESTAMP, data STRING",
+ "{ \"id\": 0, \"ts\": \"2023-08-01T09:20:44.294658+00:00\", \"data\": \"new-data-0\"}\n"
+ + "{ \"id\": 5, \"ts\": \"2023-08-02T09:23:44.294658+00:00\", \"data\": \"new-data-5\"}\n");
+
+ String date = "2017-11-22";
+ String resultSql =
+ sqlFormat("SELECT id, ts, IF(id = 0, 'new-data-0', data) FROM %s ORDER BY id", tableName);
Review Comment:
Why is there an `IF` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1293644813
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceStaticInvoke.scala:
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.iceberg.spark.SparkSQLProperties
+import org.apache.iceberg.spark.functions.SparkFunctions
+import org.apache.iceberg.spark.source.SparkTable
+import org.apache.spark.sql.catalyst.SQLConfHelper
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke
+import org.apache.spark.sql.catalyst.planning.RewrittenRowLevelCommand
+import org.apache.spark.sql.catalyst.planning.ScanOperation
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.NoStatsUnaryNode
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceIcebergData
+import org.apache.spark.sql.catalyst.plans.logical.RowLevelCommand
+import org.apache.spark.sql.catalyst.plans.logical.WriteDelta
+import org.apache.spark.sql.catalyst.plans.logical.WriteIcebergDelta
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.connector.catalog.Table
+import org.apache.spark.sql.connector.catalog.functions.ScalarFunction
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation
+import org.apache.spark.sql.execution.datasources.v2.UnplannedGroupBasedMergeOperation
+import org.apache.spark.sql.types.StructField
+import org.apache.spark.sql.types.StructType
+import scala.annotation.tailrec
+
+/**
+ * Spark analyzes the Iceberg system function to {@link StaticInvoke} which could not be pushed
+ * down to datasource. This rule will replace {@link StaticInvoke} to
+ * {@link ApplyFunctionExpression} for Iceberg system function in a filter condition.
+ */
+object ReplaceStaticInvoke extends Rule[LogicalPlan] with SQLConfHelper {
Review Comment:
Changed to the original implementation. Replacing the `StaticInvoke` in a filter condition to `ApplyFunctionExpression`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1293646978
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java:
##########
@@ -63,4 +63,9 @@ private SparkSQLProperties() {}
// Controls the WAP branch used for write-audit-publish workflow.
// When set, new snapshots will be committed to this branch.
public static final String WAP_BRANCH = "spark.wap.branch";
+
+ // Controls whether to push down Iceberg system function
+ public static final String SYSTEM_FUNC_PUSH_DOWN_ENABLED =
Review Comment:
Changed implementation. The config is set to true by default. And keep it for easy testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dramaticlly commented on pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#issuecomment-1718839226
> Thanks @rdblue for your review. Comments have been addressed.
>
> > For next steps, I think it would be good to take a look at individual row-level plans and see which ones benefit from this change. Then we can plan how to make the others work.
>
> Yeah, I think the `delete` command should be a good next step. Also, we could support the `rewrite_data_files` pass a filter containing system function after this.
hey @ConeyLiu , I did my attempt in #8560 trying to support such v2filter in rewrite_data_files, do you want to take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1286017564
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/source/PlanUtils.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
Review Comment:
Put here because `SparkBatchQueryScan` is package private.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#issuecomment-1706681435
Thanks @rdblue for your suggestions. Reverted the changes for row-level operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#issuecomment-1713758630
Thanks @rdblue for your reviewing. Comments have been addressed.
> For next steps, I think it would be good to take a look at individual row-level plans and see which ones benefit from this change. Then we can plan how to make the others work.
Yeah, I think the `delete` command should be a good next step. Also, we could support the `rewrite_data_files` pass a filter containing system function after this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1320855781
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceStaticInvoke.scala:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.iceberg.spark.functions.SparkFunctions
+import org.apache.spark.sql.catalyst.SQLConfHelper
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.FILTER
+import org.apache.spark.sql.connector.catalog.functions.ScalarFunction
+import org.apache.spark.sql.types.StructField
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Spark analyzes the Iceberg system function to {@link StaticInvoke} which could not be pushed
+ * down to datasource. This rule will replace {@link StaticInvoke} to
+ * {@link ApplyFunctionExpression} for Iceberg system function in a filter condition.
+ */
+object ReplaceStaticInvoke extends Rule[LogicalPlan] with SQLConfHelper {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.transformWithPruning (
+ _.containsAnyPattern(FILTER)) {
Review Comment:
Nit: style looks a little odd with this on a separate line. I'd prefer moving `plan.transformWithPruning(` to the same line as `_.containsAnyPattern(FILTER))`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1290876789
##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetBloomRowGroupFilter.java:
##########
@@ -334,5 +335,10 @@ private <T> boolean shouldRead(
return ROWS_MIGHT_MATCH;
}
}
+
+ @Override
+ public <T> Boolean handleNonReference(Bound<T> term) {
+ return ROWS_MIGHT_MATCH;
Review Comment:
Thanks, rebased
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285273364
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/FoldConstantSystemFunction.scala:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.iceberg.spark.functions.UnboundFunctionWrapper.SystemFunctionWrapper
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression
+import org.apache.spark.sql.catalyst.expressions.EmptyRow
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.plans.logical.LocalRelation
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * Folding those system function that applies a constant value.
+ */
+object FoldConstantSystemFunction extends Rule[LogicalPlan] {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+ case logicalPlan: LogicalPlan =>
+ logicalPlan transformExpressions {
+ case applyFunc: ApplyFunctionExpression
+ if applyFunc.function.isInstanceOf[SystemFunctionWrapper[_]] =>
+ val fakeProjectList = applyFunc.children.zipWithIndex.map(pair => Alias(pair._1, s"fake_${pair._2}")())
+ val fakePlan = Project(fakeProjectList, LocalRelation(Nil))
+ val foldedExprs =
+ ConstantFolding.apply(fakePlan).asInstanceOf[Project].projectList.map(_.asInstanceOf[Alias].child)
Review Comment:
Looks like this is because `ApplyFunctionExpression` doesn't override `foldable`.
I think we could fix that by adding this upstream:
```scala
override def foldable: Boolean = deterministic && children.forall(_.foldable)
```
`StaticInvoke` already has `foldable` implemented by its `InvokeLike` parent:
```scala
override def foldable: Boolean =
children.forall(_.foldable) && deterministic && trustedSerializable(dataType)
```
That's basically the same, except that `trustedSerializable` checks that there isn't an `ObjectType` in `dataType`, which is the output type. I'm fairly sure that will be `true`, so I'm not sure why `StaticInvoke` is not already folded. We'll have to do some debugging to track it down.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285270815
##########
spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/sql/TestSparkBucketFunction.java:
##########
@@ -351,6 +350,68 @@ public void testThatMagicFunctionsAreInvoked() {
"staticinvoke(class org.apache.iceberg.spark.functions.BucketFunction$BucketBinary");
}
+ @Test
+ public void testAnalyzedToApplyFunctionExpression() {
+ Assumptions.assumeThat(systemFunctionPushDownEnabled).isTrue();
+ // TinyInt
+ Assertions.assertThat(scalarSql("EXPLAIN EXTENDED SELECT system.bucket(5, 6Y)"))
+ .asString()
+ .isNotNull()
+ .contains("applyfunctionexpression(Wrapper(iceberg.bucket(int))");
Review Comment:
I don't think these tests are needed if we don't trick Spark into not using `StaticInvoke`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1286547442
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java:
##########
@@ -63,4 +63,9 @@ private SparkSQLProperties() {}
// Controls the WAP branch used for write-audit-publish workflow.
// When set, new snapshots will be committed to this branch.
public static final String WAP_BRANCH = "spark.wap.branch";
+
+ // Controls whether to push down Iceberg system function
+ public static final String SYSTEM_FUNC_PUSH_DOWN_ENABLED =
Review Comment:
#7886 is the first part code. It changes filters to V2 filters and adds support to convert system functions wrapped in Spark UserDefinedScalarFunc to Iceberg expressions.
Actually, I have mentioned in the description: https://github.com/apache/iceberg/pull/7886#issue-1770844392. Maybe the description is not clearly enough at #7886.
I have tried combining them in one patch. And split into this based on the https://github.com/apache/iceberg/pull/7886#discussion_r1264197758
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285542180
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/FoldConstantSystemFunction.scala:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.iceberg.spark.functions.UnboundFunctionWrapper.SystemFunctionWrapper
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression
+import org.apache.spark.sql.catalyst.expressions.EmptyRow
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.plans.logical.LocalRelation
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * Folding those system function that applies a constant value.
+ */
+object FoldConstantSystemFunction extends Rule[LogicalPlan] {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+ case logicalPlan: LogicalPlan =>
+ logicalPlan transformExpressions {
+ case applyFunc: ApplyFunctionExpression
+ if applyFunc.function.isInstanceOf[SystemFunctionWrapper[_]] =>
+ val fakeProjectList = applyFunc.children.zipWithIndex.map(pair => Alias(pair._1, s"fake_${pair._2}")())
Review Comment:
The project list of `Project` should be `NamedExpression`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#issuecomment-1668037735
Thanks @rdblue for the review.
> This should not trick Spark into not using StaticInvoke. We should find a way to support both expressions.
Explained [here](https://github.com/apache/iceberg/pull/8088#discussion_r1285622310)
> There are unrelated changes that should be moved to separate PRs so that those are reviewed in isolation.
Split out those changes to separate PRs.
> It isn't clear to me why ConstantFolding isn't already applied by the optimizer. If it isn't, then I think we should fix this in Spark and consider whether it's a good idea to do this in Iceberg. We try to only add code that extends Spark and not code that fixes Spark bugs.
The extended constant folding is just for `ApplyFunctionExpression`. We should fix it on the Spark side. It is a temporary solution for Iceberg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1290416808
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/RowLevelCommandScanRelationPushDown.scala:
##########
@@ -122,16 +130,15 @@ object RowLevelCommandScanRelationPushDown extends Rule[LogicalPlan] with Predic
private def pushFilters(
cond: Expression,
scanBuilder: ScanBuilder,
- tableAttrs: Seq[AttributeReference]): (Seq[Filter], Seq[Predicate]) = {
+ tableAttrs: Seq[AttributeReference]): (Either[Seq[Filter], Seq[Predicate]], Seq[Expression]) = {
val tableAttrSet = AttributeSet(tableAttrs)
val filters = splitConjunctivePredicates(cond).filter(_.references.subsetOf(tableAttrSet))
val normalizedFilters = DataSourceStrategy.normalizeExprs(filters, tableAttrs)
val (_, normalizedFiltersWithoutSubquery) =
normalizedFilters.partition(SubqueryExpression.hasSubquery)
- val (pushedFilters, _) = PushDownUtils.pushFilters(scanBuilder, normalizedFiltersWithoutSubquery)
- (pushedFilters.left.getOrElse(Seq.empty), pushedFilters.right.getOrElse(Seq.empty))
+ PushDownUtils.pushFilters(scanBuilder, normalizedFiltersWithoutSubquery)
Review Comment:
This was merged.
##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetBloomRowGroupFilter.java:
##########
@@ -334,5 +335,10 @@ private <T> boolean shouldRead(
return ROWS_MIGHT_MATCH;
}
}
+
+ @Override
+ public <T> Boolean handleNonReference(Bound<T> term) {
+ return ROWS_MIGHT_MATCH;
Review Comment:
Both of these were merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285268752
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java:
##########
@@ -1304,6 +1313,82 @@ public void testDeleteToCustomWapBranchWithoutWhereClause() throws NoSuchTableEx
});
}
+ @Test
+ public void testDeleteFilterWithSystemFuncOnPartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createPartitionedTable(spark, tableName, "years(ts)", tableProperties);
+
+ String date = "2018-12-21";
+ String remainSql =
+ sqlFormat(
+ "SELECT * FROM %s WHERE %s.system.years(ts) != %s.system.years(date('%s')) ORDER BY id",
+ tableName, catalogName, catalogName, date);
+
+ List<Object[]> expected = sql(remainSql);
+
+ withSQLConf(
+ ImmutableMap.of(SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED, "true"),
+ () -> {
+ String deleteSql =
+ sqlFormat(
+ "DELETE FROM %s WHERE %s.system.years(ts) == %s.system.years(date('%s'))",
+ tableName, catalogName, catalogName, date);
+ Object explain = scalarSql("EXPLAIN EXTENDED %s", deleteSql);
+ // Metadata delete
+ Assertions.assertThat((String) explain).contains("DeleteFromTableWithFilters");
+ Assertions.assertThat((String) explain).contains("[years(ts) = " + years(date));
+
+ sql(deleteSql);
+
+ List<Object[]> actual = sql(remainSql);
Review Comment:
This should be `SELECT *` without a `WHERE` clause. Otherwise this does not check that the rows were actually removed because they would be ignored by this SQL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1286438766
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java:
##########
@@ -1304,6 +1313,82 @@ public void testDeleteToCustomWapBranchWithoutWhereClause() throws NoSuchTableEx
});
}
+ @Test
+ public void testDeleteFilterWithSystemFuncOnPartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createPartitionedTable(spark, tableName, "years(ts)", tableProperties);
+
+ String date = "2018-12-21";
+ String remainSql =
+ sqlFormat(
+ "SELECT * FROM %s WHERE %s.system.years(ts) != %s.system.years(date('%s')) ORDER BY id",
Review Comment:
It's fine to have a test that the function is foldable, but I don't think it should be used to produce expected rows. Instead, use a constant here and test that the folding is done elsewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1290459916
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java:
##########
@@ -63,4 +63,9 @@ private SparkSQLProperties() {}
// Controls the WAP branch used for write-audit-publish workflow.
// When set, new snapshots will be committed to this branch.
public static final String WAP_BRANCH = "spark.wap.branch";
+
+ // Controls whether to push down Iceberg system function
+ public static final String SYSTEM_FUNC_PUSH_DOWN_ENABLED =
Review Comment:
Okay, I understand. Thanks for explaining. I thought this was about constant folding, but it's really two issues. `StaticInvoke` can't be converted to a v2 filter and using `ApplyFunctionExpression` instead breaks constant folding because that expression is not `foldable`.
I don't like introducing more problems, so it's unfortunate that this breaks constant folding. I also would really like to avoid adding a configuration that chooses between `StaticInvoke` and `ApplyFunctionExpression`. I think it may be cleaner to choose when to convert `StaticInvoke` rather than using a wrapper. Converting in a rule would allow us to check whether the `StaticInvoke` expression would be folded and leave it alone. We could also be more targeted in a rule and only convert `StaticInvoke` expressions that are in predicates that could be pushed down. An expression in a `Project` should be left alone.
There could be a problem with that approach, which is that Spark might apply a rule that keeps convert back to `StaticInvoke`. If needed, maybe we could inject a wrapper at that point, or create a subclass of `ApplyFunctionExpression` that doesn't match the rule.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1316515701
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceStaticInvoke.scala:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.iceberg.spark.SparkSQLProperties
+import org.apache.iceberg.spark.functions.SparkFunctions
+import org.apache.iceberg.spark.source.SparkTable
+import org.apache.spark.sql.catalyst.SQLConfHelper
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.connector.catalog.Table
+import org.apache.spark.sql.connector.catalog.functions.ScalarFunction
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation
+import org.apache.spark.sql.types.StructField
+import org.apache.spark.sql.types.StructType
+import scala.annotation.tailrec
+
+/**
+ * Spark analyzes the Iceberg system function to {@link StaticInvoke} which could not be pushed
+ * down to datasource. This rule will replace {@link StaticInvoke} to
+ * {@link ApplyFunctionExpression} for Iceberg system function in a filter condition.
+ */
+object ReplaceStaticInvoke extends Rule[LogicalPlan] with SQLConfHelper {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ if (systemFuncPushDownEnabled()) {
+ rewrite(plan)
+ } else {
+ plan
+ }
+ }
+
+ private def rewrite(plan: LogicalPlan): LogicalPlan = plan transform {
+ case filter @ Filter(condition, child) if icebergRelation(child) =>
+ val newCondition = replaceStaticInvoke(condition)
+ if (newCondition fastEquals condition) {
+ filter
+ } else {
+ filter.copy(condition = newCondition)
+ }
+ }
+
+ private def systemFuncPushDownEnabled(): Boolean = {
+ conf.getConfString(
+ SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED,
+ SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED_DEFAULT.toString
+ ).toBoolean
+ }
+
+ private def replaceStaticInvoke(expression: Expression): Expression = expression.transform {
+ case invoke: StaticInvoke =>
+ invoke.functionName match {
+ case "invoke" =>
+ if (invoke.arguments.forall(_.foldable)) {
+ // The invoke should be folded into constant
+ return invoke
+ }
+
+ // Adaptive from `resolveV2Function` in org.apache.spark.sql.catalyst.analysis.ResolveFunctions
+ val unbound = SparkFunctions.loadFunctionByClass(invoke.staticObject)
+ if (unbound == null) {
+ return invoke
+ }
+
+ val inputType = StructType(invoke.arguments.zipWithIndex.map {
+ case (exp, pos) => StructField(s"_$pos", exp.dataType, exp.nullable)
+ })
+
+ val bound = try {
+ unbound.bind(inputType)
+ } catch {
+ case _: Exception =>
+ return invoke
+ }
+
+ if (bound.inputTypes().length != invoke.arguments.length) {
+ return invoke
+ }
+
+ bound match {
+ case scalarFunc: ScalarFunction[_] =>
+ ApplyFunctionExpression(scalarFunc, invoke.arguments)
+ case _ => invoke
+ }
+ }
+ }
+
+ @tailrec
+ private def icebergRelation(plan: LogicalPlan): Boolean = {
+
+ def isIcebergTable(table: Table): Boolean = table match {
+ case _: SparkTable => true
+ case _ => false
+ }
+
+ plan match {
+ case relation: DataSourceV2Relation =>
+ isIcebergTable(relation.table)
+ case scanRelation: DataSourceV2ScanRelation =>
+ icebergRelation(scanRelation.relation)
Review Comment:
I don't think it would make sense to convert when there is a scan relation because that means the pushdown already occurred, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#issuecomment-1716660808
Thanks, @ConeyLiu! Glad to get this in before the 1.4 release!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue merged PR #8088:
URL: https://github.com/apache/iceberg/pull/8088
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#issuecomment-1716878192
Thanks @rdblue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285624053
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/functions/UnboundFunctionWrapper.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.functions;
+
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.connector.catalog.functions.BoundFunction;
+import org.apache.spark.sql.connector.catalog.functions.ScalarFunction;
+import org.apache.spark.sql.connector.catalog.functions.UnboundFunction;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructType;
+
+/**
+ * An UnboundFunction wrapper that wraps a system function. The bound function by this wrapper will
+ * hide the magic method.
+ */
+public class UnboundFunctionWrapper implements UnboundFunction {
Review Comment:
See the comments at [here](https://github.com/apache/iceberg/pull/8088/files#r1285622310)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285268788
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java:
##########
@@ -1304,6 +1313,82 @@ public void testDeleteToCustomWapBranchWithoutWhereClause() throws NoSuchTableEx
});
}
+ @Test
+ public void testDeleteFilterWithSystemFuncOnPartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createPartitionedTable(spark, tableName, "years(ts)", tableProperties);
+
+ String date = "2018-12-21";
+ String remainSql =
+ sqlFormat(
+ "SELECT * FROM %s WHERE %s.system.years(ts) != %s.system.years(date('%s')) ORDER BY id",
+ tableName, catalogName, catalogName, date);
+
+ List<Object[]> expected = sql(remainSql);
+
+ withSQLConf(
+ ImmutableMap.of(SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED, "true"),
+ () -> {
+ String deleteSql =
+ sqlFormat(
+ "DELETE FROM %s WHERE %s.system.years(ts) == %s.system.years(date('%s'))",
+ tableName, catalogName, catalogName, date);
+ Object explain = scalarSql("EXPLAIN EXTENDED %s", deleteSql);
+ // Metadata delete
+ Assertions.assertThat((String) explain).contains("DeleteFromTableWithFilters");
+ Assertions.assertThat((String) explain).contains("[years(ts) = " + years(date));
+
+ sql(deleteSql);
+
+ List<Object[]> actual = sql(remainSql);
+ assertEquals("Results should match", expected, actual);
+ });
+ }
+
+ @Test
+ public void testDeleteFilterWithSystemFuncOnUnpartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createUnpartitionedTable(spark, tableName, tableProperties);
+
+ String date = "2018-12-21";
+ String remainSql =
+ sqlFormat(
+ "SELECT * FROM %s WHERE %s.system.months(ts) = %s.system.months(date('%s')) ORDER BY id",
+ tableName, catalogName, catalogName, date);
+
+ List<Object[]> expected = sql(remainSql);
+
+ withSQLConf(
+ ImmutableMap.of(SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED, "true"),
+ () -> {
+ String deleteSql =
+ sqlFormat(
+ "DELETE FROM %s WHERE %s.system.months(ts) != %s.system.months(date('%s'))",
+ tableName, catalogName, catalogName, date);
+ Object explain = scalarSql("EXPLAIN EXTENDED %s", deleteSql);
+ Assertions.assertThat((String) explain)
+ .contains("Filter NOT (applyfunctionexpression(Wrapper(iceberg.months(timestamp))");
+ Assertions.assertThat((String) explain).contains("[filters=NOT (ts = " + months(date));
+
+ sql(deleteSql);
+
+ List<Object[]> actual = sql(remainSql);
+ assertEquals("Results should match", expected, actual);
Review Comment:
This has the same problems as the test case above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285269790
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestSystemFunctionPushDownDQL.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.extensions;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.createPartitionedTable;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.createUnpartitionedTable;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.days;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.hours;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.months;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.years;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.spark.SparkCatalogConfig;
+import org.apache.iceberg.spark.SparkSQLProperties;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.assertj.core.api.Assertions;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runners.Parameterized;
+
+public class TestSystemFunctionPushDownDQL extends SparkExtensionsTestBase {
+ private final String fileFormat;
+
+ public TestSystemFunctionPushDownDQL(
Review Comment:
It looks like this is intended to test file filter pushdown that should be in a separate PR, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285268131
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/RowLevelCommandScanRelationPushDown.scala:
##########
@@ -122,16 +130,15 @@ object RowLevelCommandScanRelationPushDown extends Rule[LogicalPlan] with Predic
private def pushFilters(
cond: Expression,
scanBuilder: ScanBuilder,
- tableAttrs: Seq[AttributeReference]): (Seq[Filter], Seq[Predicate]) = {
+ tableAttrs: Seq[AttributeReference]): (Either[Seq[Filter], Seq[Predicate]], Seq[Expression]) = {
val tableAttrSet = AttributeSet(tableAttrs)
val filters = splitConjunctivePredicates(cond).filter(_.references.subsetOf(tableAttrSet))
val normalizedFilters = DataSourceStrategy.normalizeExprs(filters, tableAttrs)
val (_, normalizedFiltersWithoutSubquery) =
normalizedFilters.partition(SubqueryExpression.hasSubquery)
- val (pushedFilters, _) = PushDownUtils.pushFilters(scanBuilder, normalizedFiltersWithoutSubquery)
- (pushedFilters.left.getOrElse(Seq.empty), pushedFilters.right.getOrElse(Seq.empty))
+ PushDownUtils.pushFilters(scanBuilder, normalizedFiltersWithoutSubquery)
Review Comment:
This looks like an unrelated bug. Should we fix it in a different PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285270602
##########
spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/SystemFunctionPushDownHelper.java:
##########
@@ -29,6 +31,21 @@ public static void createUnpartitionedTable(SparkSession spark, String tableName
insertRecords(spark, tableName);
}
+ public static void createUnpartitionedTable(
+ SparkSession spark, String tableName, Map<String, String> props) {
+ if (props.isEmpty()) {
+ createUnpartitionedTable(spark, tableName);
+ } else {
+ String tblProps = Joiner.on("','").withKeyValueSeparator("'='").join(props);
+ sql(
+ spark,
+ "CREATE TABLE %s (id BIGINT, ts TIMESTAMP, data STRING) USING iceberg TBLPROPERTIES ('%s')",
Review Comment:
Isn't this SQL invalid if there are no table properties because it would result in `TBLPROPERTIES ('')`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285270063
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java:
##########
@@ -63,4 +63,9 @@ private SparkSQLProperties() {}
// Controls the WAP branch used for write-audit-publish workflow.
// When set, new snapshots will be committed to this branch.
public static final String WAP_BRANCH = "spark.wap.branch";
+
+ // Controls whether to push down Iceberg system function
+ public static final String SYSTEM_FUNC_PUSH_DOWN_ENABLED =
Review Comment:
I don't think it is clear why this is necessary. Why is this needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1285982024
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java:
##########
@@ -2728,6 +2736,105 @@ public void testMergeToWapBranchWithTableBranchIdentifier() {
branch)));
}
+ @Test
+ public void testMergeFilterWithSystemFunctionsOnPartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createPartitionedTable(spark, tableName, "years(ts)", tableProperties);
+
+ createOrReplaceView(
+ "source",
+ "id BIGINT, ts TIMESTAMP, data STRING",
+ "{ \"id\": 0, \"ts\": \"2023-08-01T09:20:44.294658+00:00\", \"data\": \"new-data-0\"}\n"
+ + "{ \"id\": 5, \"ts\": \"2023-08-02T09:23:44.294658+00:00\", \"data\": \"new-data-5\"}\n");
+
+ String date = "2017-11-22";
+ String resultSql =
+ sqlFormat("SELECT id, ts, IF(id = 0, 'new-data-0', data) FROM %s ORDER BY id", tableName);
+ List<Object[]> expected = sql(resultSql);
+
+ withSQLConf(
+ ImmutableMap.of(SparkSQLProperties.SYSTEM_FUNC_PUSH_DOWN_ENABLED, "true"),
+ () -> {
+ String mergeSql =
+ sqlFormat(
+ "MERGE INTO %s AS t USING source AS s "
+ + "ON t.id = s.id AND %s.system.years(t.ts) = %s.system.years(date('%s')) "
+ + "WHEN MATCHED "
+ + " THEN UPDATE SET t.data = s.data "
+ + "WHEN NOT MATCHED AND s.id != 5 "
+ + " THEN INSERT *",
+ tableName, catalogName, catalogName, date);
+ Dataset<Row> df = spark.sql(mergeSql);
+ // Should not contain applyfunction since filter function could be evaluated completely on
+ // Iceberg side and will not return post scan filters
+ Assertions.assertThat(df.queryExecution().sparkPlan().toString())
+ .doesNotContain("Filter (applyfunctionexpression(Wrapper(iceberg.years(timestamp))");
Review Comment:
Sure. The `TestSystemFunctionPushDownDQL` part has changed. Row-level commands will change later because they are more complicated than batch san.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1286960408
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestSystemFunctionPushDownDQL.java:
##########
@@ -0,0 +1,441 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.extensions;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT;
+import static org.apache.iceberg.expressions.Expressions.bucket;
+import static org.apache.iceberg.expressions.Expressions.day;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.hour;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.month;
+import static org.apache.iceberg.expressions.Expressions.notEqual;
+import static org.apache.iceberg.expressions.Expressions.truncate;
+import static org.apache.iceberg.expressions.Expressions.year;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.STRUCT;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.createPartitionedTable;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.createUnpartitionedTable;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.days;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.hours;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.months;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.years;
+
+import java.sql.Date;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.expressions.ExpressionUtil;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.spark.SparkCatalogConfig;
+import org.apache.iceberg.spark.SparkSQLProperties;
+import org.apache.iceberg.spark.functions.UnboundFunctionWrapper;
+import org.apache.iceberg.spark.source.PlanUtils;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression;
+import org.apache.spark.sql.catalyst.expressions.Expression;
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke;
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan;
+import org.assertj.core.api.Assertions;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runners.Parameterized;
+
+public class TestSystemFunctionPushDownDQL extends SparkExtensionsTestBase {
+ private final String fileFormat;
+
+ public TestSystemFunctionPushDownDQL(
+ String catalogName, String implementation, Map<String, String> config, String fileFormat) {
+ super(catalogName, implementation, config);
+ this.fileFormat = fileFormat;
+ }
+
+ @Parameterized.Parameters(
+ name = "catalogName = {0}, implementation = {1}, config = {2}, fileFormat = {3}")
+ public static Object[][] parameters() {
+ return new Object[][] {
+ {
+ SparkCatalogConfig.HIVE.catalogName(),
+ SparkCatalogConfig.HIVE.implementation(),
+ SparkCatalogConfig.HIVE.properties(),
+ "parquet"
+ },
+ {
+ SparkCatalogConfig.HIVE.catalogName(),
+ SparkCatalogConfig.HIVE.implementation(),
+ SparkCatalogConfig.HIVE.properties(),
+ "orc"
+ },
+ {
+ SparkCatalogConfig.HIVE.catalogName(),
+ SparkCatalogConfig.HIVE.implementation(),
+ SparkCatalogConfig.HIVE.properties(),
+ "avro"
+ },
+ };
+ }
+
+ @Before
+ public void before() {
+ sql("USE %s", catalogName);
+ }
+
+ @After
+ public void removeTables() {
+ sql("DROP TABLE IF EXISTS %s", tableName);
+ }
+
+ @Test
+ public void testFoldingConstantExpression() {
Review Comment:
Checking the constant folding here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1282915327
##########
orc/src/main/java/org/apache/iceberg/orc/ExpressionToSearchArgument.java:
##########
@@ -270,7 +271,8 @@ public <T> Action notStartsWith(Bound<T> expr, Literal<T> lit) {
@Override
public <T> Action predicate(BoundPredicate<T> pred) {
- if (UNSUPPORTED_TYPES.contains(pred.ref().type().typeId())) {
+ if (UNSUPPORTED_TYPES.contains(pred.ref().type().typeId())
+ || !(pred.term() instanceof BoundReference)) {
Review Comment:
Should evaluate on reference term only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1282955480
##########
orc/src/main/java/org/apache/iceberg/orc/ExpressionToSearchArgument.java:
##########
@@ -270,7 +271,8 @@ public <T> Action notStartsWith(Bound<T> expr, Literal<T> lit) {
@Override
public <T> Action predicate(BoundPredicate<T> pred) {
- if (UNSUPPORTED_TYPES.contains(pred.ref().type().typeId())) {
+ if (UNSUPPORTED_TYPES.contains(pred.ref().type().typeId())
+ || !(pred.term() instanceof BoundReference)) {
Review Comment:
The reason is that we support push-down transform functions now. However, the file format can't handle such filters. The reason for parquet too. Another choice is we don't push down those transform functions for unpartitioned tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1290876896
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java:
##########
@@ -1304,6 +1322,97 @@ public void testDeleteToCustomWapBranchWithoutWhereClause() throws NoSuchTableEx
});
}
+ @Test
+ public void testDeleteFilterWithSystemFuncOnPartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createPartitionedTable(spark, tableName, "years(ts)", tableProperties);
+
+ int targetYears = years("2018-12-21");
Review Comment:
Sure, will change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Wrap system functions to allow filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1290807308
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java:
##########
@@ -1304,6 +1322,97 @@ public void testDeleteToCustomWapBranchWithoutWhereClause() throws NoSuchTableEx
});
}
+ @Test
+ public void testDeleteFilterWithSystemFuncOnPartitionedTable() {
+ Assumptions.assumeThat(catalogName).isNotEqualTo("spark_catalog");
+
+ Map<String, String> tableProperties = Maps.newHashMap();
+ tableProperties.put(DEFAULT_FILE_FORMAT, fileFormat);
+ tableProperties.put(WRITE_DISTRIBUTION_MODE, distributionMode);
+ tableProperties.put(SPARK_WRITE_PARTITIONED_FANOUT_ENABLED, String.valueOf(fanoutEnabled));
+ tableProperties.putAll(extraTableProperties());
+ createPartitionedTable(spark, tableName, "years(ts)", tableProperties);
+
+ int targetYears = years("2018-12-21");
Review Comment:
It seems odd that this is a date string. Why not a timestamp? It would be better to rename this to `dateStrToYearOrdinal` or something more clear about the format of both the input string and the output value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1294838426
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceStaticInvoke.scala:
##########
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.iceberg.spark.SparkSQLProperties
+import org.apache.iceberg.spark.functions.SparkFunctions
+import org.apache.iceberg.spark.source.SparkTable
+import org.apache.spark.sql.catalyst.SQLConfHelper
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceIcebergData
+import org.apache.spark.sql.catalyst.plans.logical.RowLevelCommand
+import org.apache.spark.sql.catalyst.plans.logical.RowLevelWrite
+import org.apache.spark.sql.catalyst.plans.logical.WriteDelta
+import org.apache.spark.sql.catalyst.plans.logical.WriteIcebergDelta
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.connector.catalog.Table
+import org.apache.spark.sql.connector.catalog.functions.ScalarFunction
+import org.apache.spark.sql.connector.write.RowLevelOperation
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation
+import org.apache.spark.sql.types.StructField
+import org.apache.spark.sql.types.StructType
+import scala.annotation.tailrec
+
+/**
+ * Spark analyzes the Iceberg system function to {@link StaticInvoke} which could not be pushed
+ * down to datasource. This rule will replace {@link StaticInvoke} to
+ * {@link ApplyFunctionExpression} for Iceberg system function in a filter condition.
+ */
+object ReplaceStaticInvoke extends Rule[LogicalPlan] with SQLConfHelper {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ if (systemFuncPushDownEnabled()) {
+ rewrite(plan)
+ } else {
+ plan
+ }
+ }
+
+ private def rewrite(plan: LogicalPlan): LogicalPlan = plan transform {
Review Comment:
@rdblue I have tried to make it more simply but there is still some case to handle for row-level command. Because the condition is not only contained in the `Filter`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1320855027
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceStaticInvoke.scala:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.iceberg.spark.functions.SparkFunctions
+import org.apache.spark.sql.catalyst.SQLConfHelper
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.FILTER
+import org.apache.spark.sql.connector.catalog.functions.ScalarFunction
+import org.apache.spark.sql.types.StructField
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Spark analyzes the Iceberg system function to {@link StaticInvoke} which could not be pushed
+ * down to datasource. This rule will replace {@link StaticInvoke} to
+ * {@link ApplyFunctionExpression} for Iceberg system function in a filter condition.
+ */
+object ReplaceStaticInvoke extends Rule[LogicalPlan] with SQLConfHelper {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.transformWithPruning (
+ _.containsAnyPattern(FILTER)) {
+
+ case filter @ Filter(condition, _) =>
Review Comment:
I think it may be useful to further narrow down the matching expressions by finding conditions that include a subclass of `BinaryComparison`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1320856034
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/source/PlanUtils.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.source;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan;
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation;
+import scala.PartialFunction;
+import scala.collection.JavaConverters;
+import scala.collection.Seq;
+
+public class PlanUtils {
+ private PlanUtils() {}
+
+ public static List<Expression> getScanPushDownFilters(LogicalPlan logicalPlan) {
Review Comment:
Iceberg does not use `get` in function names. Use a more descriptive verb.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1321446880
##########
spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceStaticInvoke.scala:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.iceberg.spark.functions.SparkFunctions
+import org.apache.spark.sql.catalyst.SQLConfHelper
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.FILTER
+import org.apache.spark.sql.connector.catalog.functions.ScalarFunction
+import org.apache.spark.sql.types.StructField
+import org.apache.spark.sql.types.StructType
+
+/**
+ * Spark analyzes the Iceberg system function to {@link StaticInvoke} which could not be pushed
+ * down to datasource. This rule will replace {@link StaticInvoke} to
+ * {@link ApplyFunctionExpression} for Iceberg system function in a filter condition.
+ */
+object ReplaceStaticInvoke extends Rule[LogicalPlan] with SQLConfHelper {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.transformWithPruning (
+ _.containsAnyPattern(FILTER)) {
Review Comment:
OK, updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#discussion_r1320860311
##########
spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestSystemFunctionPushDownDQL.java:
##########
@@ -0,0 +1,338 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.extensions;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT;
+import static org.apache.iceberg.expressions.Expressions.bucket;
+import static org.apache.iceberg.expressions.Expressions.day;
+import static org.apache.iceberg.expressions.Expressions.equal;
+import static org.apache.iceberg.expressions.Expressions.greaterThan;
+import static org.apache.iceberg.expressions.Expressions.greaterThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.hour;
+import static org.apache.iceberg.expressions.Expressions.lessThan;
+import static org.apache.iceberg.expressions.Expressions.lessThanOrEqual;
+import static org.apache.iceberg.expressions.Expressions.month;
+import static org.apache.iceberg.expressions.Expressions.notEqual;
+import static org.apache.iceberg.expressions.Expressions.truncate;
+import static org.apache.iceberg.expressions.Expressions.year;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.STRUCT;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.createPartitionedTable;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.createUnpartitionedTable;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.timestampStrToDayOrdinal;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.timestampStrToHourOrdinal;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.timestampStrToMonthOrdinal;
+import static org.apache.iceberg.spark.SystemFunctionPushDownHelper.timestampStrToYearOrdinal;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.expressions.ExpressionUtil;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.spark.SparkCatalogConfig;
+import org.apache.iceberg.spark.source.PlanUtils;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.catalyst.expressions.ApplyFunctionExpression;
+import org.apache.spark.sql.catalyst.expressions.Expression;
+import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke;
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan;
+import org.assertj.core.api.Assertions;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runners.Parameterized;
+
+public class TestSystemFunctionPushDownDQL extends SparkExtensionsTestBase {
+ private final String fileFormat;
+
+ public TestSystemFunctionPushDownDQL(
+ String catalogName, String implementation, Map<String, String> config, String fileFormat) {
+ super(catalogName, implementation, config);
+ this.fileFormat = fileFormat;
+ }
+
+ @Parameterized.Parameters(
+ name = "catalogName = {0}, implementation = {1}, config = {2}, fileFormat = {3}")
+ public static Object[][] parameters() {
+ return new Object[][] {
+ {
+ SparkCatalogConfig.HIVE.catalogName(),
+ SparkCatalogConfig.HIVE.implementation(),
+ SparkCatalogConfig.HIVE.properties(),
+ "parquet"
Review Comment:
I implemented this: https://github.com/ConeyLiu/iceberg/pull/1
I was testing locally and went ahead with the change to be able to do some debugging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #8088: Spark: Rule for converting StaticInvoke to ApplyFunctionExpression for V2 filter push down
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on PR #8088:
URL: https://github.com/apache/iceberg/pull/8088#issuecomment-1712935792
Thanks for changing this to just `Filter`, @ConeyLiu!
For next steps, I think it would be good to take a look at individual row-level plans and see which ones benefit from this change. Then we can plan how to make the others work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org