You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "steFaiz (via GitHub)" <gi...@apache.org> on 2026/02/11 09:00:16 UTC
[PR] [spark] supports converting some SparkPredicate to Paimon between LeafPredicate [paimon]
steFaiz opened a new pull request, #7265:
URL: https://github.com/apache/paimon/pull/7265
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
Currently, spark will convert Between predicate to the composition of lessOrEqual and GreaterOrEqual, this PR is about to recognize this pattern, converting some And CompoundedPredicate to a single Between LeafPredicate.
<!-- Linking this pull request to the issue -->
Linked issue: none
<!-- What is the purpose of the change -->
### Tests
Please see `org.apache.paimon.spark.sql.SparkV2FilterConverterTestBase`
<!-- List UT and IT cases to verify this change -->
### API and Format
No changes.
<!-- Does this change affect API or storage format -->
### Documentation
No changes.
<!-- Does this change introduce a new feature -->
### Generative AI tooling
This PR is fully hand-written.
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] supports converting some SparkPredicate to Paimon between LeafPredicate [paimon]
Posted by "steFaiz (via GitHub)" <gi...@apache.org>.
steFaiz commented on code in PR #7265:
URL: https://github.com/apache/paimon/pull/7265#discussion_r2798202820
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/util/PredicateUtils.scala:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.spark.util
+
+import org.apache.paimon.predicate.{Between, CompareUtils, GreaterOrEqual, LeafPredicate, LessOrEqual, Predicate}
+
+import java.util
+import java.util.Objects
+
+object PredicateUtils {
Review Comment:
@JingsongLi Thanks for your advise! I've refactored it. But it's worthy to note that implementing this logic in core is much more complicated than in spark. We have to consider broader scenarios, for example:
1. recursive: OR(AND(a >= 1, a <= 10, a is not null), b > 10, ... )
2. chained AND: AND(a >= 1, AND(a <= 10, b > 10)) (this could be converted to BETWEEN(a, 1, 10))
and more. maybe I've missed some scenario.
Please take a look and i'm pleasant to improve my code and fix potential bugs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] supports converting some SparkPredicate to Paimon between LeafPredicate [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #7265:
URL: https://github.com/apache/paimon/pull/7265
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] supports converting some SparkPredicate to Paimon between LeafPredicate [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #7265:
URL: https://github.com/apache/paimon/pull/7265#discussion_r2796791851
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/util/PredicateUtils.scala:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.spark.util
+
+import org.apache.paimon.predicate.{Between, CompareUtils, GreaterOrEqual, LeafPredicate, LessOrEqual, Predicate}
+
+import java.util
+import java.util.Objects
+
+object PredicateUtils {
Review Comment:
We could have a PredicateRewrite in core. Maybe invoked in `ReadBuilder.withFilter`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] supports converting some SparkPredicate to Paimon between LeafPredicate [paimon]
Posted by "steFaiz (via GitHub)" <gi...@apache.org>.
steFaiz commented on code in PR #7265:
URL: https://github.com/apache/paimon/pull/7265#discussion_r2798202820
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/util/PredicateUtils.scala:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.spark.util
+
+import org.apache.paimon.predicate.{Between, CompareUtils, GreaterOrEqual, LeafPredicate, LessOrEqual, Predicate}
+
+import java.util
+import java.util.Objects
+
+object PredicateUtils {
Review Comment:
@JingsongLi Thanks for your advise! I've refactored it. But it's worthy to note that implementing this logic in core is much more complicated than in spark. We have to consider broader scenarios, for example:
1. recursive: OR(AND(GREATER_OR_EQ, LESS_OR_EQ), ... )
2. chained AND: AND(a >= 1, AND(a <= 10, b > 10))
and more. maybe I've missed some scenario.
Please take a look and i'm pleasant to improve my code and fix potential bugs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] supports converting some SparkPredicate to Paimon between LeafPredicate [paimon]
Posted by "steFaiz (via GitHub)" <gi...@apache.org>.
steFaiz commented on PR #7265:
URL: https://github.com/apache/paimon/pull/7265#issuecomment-3884972797
This PR also rewrite data filters in PaimonBatchScanBuilder, trying to merge candidates filters into `Between` LeafPredicate.
This is because Spark will split Between into two separated LessOfEqual and GreaterOrEqual SparkPredicates (Not a AND predicate with children).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] supports converting some SparkPredicate to Paimon between LeafPredicate [paimon]
Posted by "steFaiz (via GitHub)" <gi...@apache.org>.
steFaiz commented on code in PR #7265:
URL: https://github.com/apache/paimon/pull/7265#discussion_r2798202820
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/util/PredicateUtils.scala:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.spark.util
+
+import org.apache.paimon.predicate.{Between, CompareUtils, GreaterOrEqual, LeafPredicate, LessOrEqual, Predicate}
+
+import java.util
+import java.util.Objects
+
+object PredicateUtils {
Review Comment:
@JingsongLi Thanks for your advise! I've refactored it. But it's worthy to note that implementing this logic in core is much more complicated than in spark. We have to consider broader scenarios, for example:
1. recursive: OR(AND(a >= 1, a <= 10, a is not null), b > 10, ... )
2. chained AND: AND(a >= 1, AND(a <= 10, b > 10))
and more. maybe I've missed some scenario.
Please take a look and i'm pleasant to improve my code and fix potential bugs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org