You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/10/21 14:42:07 UTC
(spark) branch branch-4.x updated: [SPARK-60397][SQL] Fix ORC filter pushdown on nanosecond timestamp columns read with a lower precision
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 75c8aeab75ff [SPARK-60397][SQL] Fix ORC filter pushdown on nanosecond timestamp columns read with a lower precision
75c8aeab75ff is described below
commit 75c8aeab75ff05f49f4177908addbcf2f1009c17
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Wed Oct 21 07:32:05 2026 -0700
[SPARK-60397][SQL] Fix ORC filter pushdown on nanosecond timestamp columns read with a lower precision
### What changes were proposed in this pull request?
This PR aims to fix the ORC filter pushdown on `TIMESTAMP_LTZ(p)` and `TIMESTAMP_NTZ(p)` columns when the ORC file was written with a higher precision than `p`.
Since SPARK-57823 (#57681), ORC pushes down the predicates on these columns. However, the ORC reader truncates the stored nanoseconds to the read precision (SPARK-57455, #56557), while ORC evaluates a pushed down predicate against the stored values. Unlike Parquet (SPARK-60380, #59641), an ORC file schema has the written precision through the Catalyst type attribute (`spark.sql.catalyst.type`), and `OrcFilters.createFilter` builds a `SearchArgument` for each file from its file schema. [...]
- `OrcTypeOps.acceptsReadType` is added (default `true`), like `ParquetFilterOps.acceptsReadType` of SPARK-60358 (#59606). The ops of `TIMESTAMP_LTZ(p)` and `TIMESTAMP_NTZ(p)` return `true` only for `TIMESTAMP_LTZ(q)` and `TIMESTAMP_NTZ(q)` with `q >= p`, respectively.
- `OrcFilters.createFilter` gets the read schema (`requiredSchema` in V1, `readDataSchema` in V2) and checks `acceptsReadType` before it converts a predicate with a value on a Types Framework field.
A foreign ORC timestamp column without the attribute cannot be read with the nanosecond timestamp types (`UNSUPPORTED_FEATURE.ORC_TYPE_CAST`), so the written precision is always known here. Since it is known only for each file, the pushed filters in the query plan are not changed.
### Why are the changes needed?
The query results are wrong. For example, `00:00:00.123456789` is read as `00:00:00.1234567` with `TIMESTAMP_NTZ(7)`, but the pushed down `=` compares the literal with the stored value.
```scala
// spark.sql.timestampNanosTypes.enabled=true
sql("SELECT CAST('1970-01-01 00:00:00.123456789' AS TIMESTAMP_NTZ(9)) AS ts UNION ALL " +
"SELECT CAST('1970-01-01 00:00:01' AS TIMESTAMP_NTZ(9))").coalesce(1).write.orc("/tmp/nanos")
val df = spark.read.schema("ts TIMESTAMP_NTZ(7)").orc("/tmp/nanos")
df.collect()
// Array([1970-01-01T00:00:00.123456700], [1970-01-01T00:00:01])
df.where("ts = CAST('1970-01-01 00:00:00.1234567' AS TIMESTAMP_NTZ(7))").collect()
// Array()
```
`<=>`, `<=` and `IN` return no rows in the same way, with both V1 and V2, with both the vectorized and the row-based readers, and for `TIMESTAMP_LTZ` too. The expected result is `Array([1970-01-01T00:00:00.123456700])`, which is the result with `spark.sql.orc.filterPushdown=false`.
### Does this PR introduce _any_ user-facing change?
Yes, this fixes the wrong query results in the unreleased branches (`master`, `branch-4.x` and `branch-4.3`), including [v4.3.0-rc1](https://github.com/apache/spark/releases/tag/v4.3.0-rc1) (2026-08-31) and [v4.3.0-rc2](https://github.com/apache/spark/releases/tag/v4.3.0-rc2) (2026-10-12). The predicates with a value are not pushed down anymore only for the files written with a higher precision than the read precision. The nanosecond timestamp types are a preview feature, which is dis [...]
### How was this patch tested?
Pass the CIs with the newly added test cases in `OrcFilterSuite` and `OrcV1FilterSuite`. The end-to-end test case fails without this fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59654 from dongjoon-hyun/SPARK-60397.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
(cherry picked from commit cd06735c008285bddb9bd976b3e931f37eb4990f)
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../execution/datasources/orc/OrcFileFormat.scala | 2 +-
.../sql/execution/datasources/orc/OrcFilters.scala | 56 +++++++++++----
.../datasources/orc/types/ops/OrcTypeOps.scala | 12 +++-
.../orc/types/ops/TimestampNanosOrcOps.scala | 24 ++++++-
.../v2/orc/OrcPartitionReaderFactory.scala | 3 +-
.../execution/datasources/orc/OrcFilterSuite.scala | 81 +++++++++++++++++++++-
6 files changed, 160 insertions(+), 18 deletions(-)
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala
index c069abe3b438..7bff46e8987a 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala
@@ -198,7 +198,7 @@ class OrcFileFormat
// ORC predicate pushdown
if (orcFilterPushDown && filters.nonEmpty) {
val fileSchema = OrcUtils.toCatalystSchema(orcSchema)
- OrcFilters.createFilter(fileSchema, filters).foreach { f =>
+ OrcFilters.createFilter(fileSchema, filters, Some(requiredSchema)).foreach { f =>
OrcInputFormat.setSearchArgument(conf, f, fileSchema.fieldNames)
}
}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilters.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilters.scala
index 0e389ba99f15..2d129b2ca8a1 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilters.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilters.scala
@@ -68,11 +68,29 @@ private[sql] object OrcFilters extends OrcFiltersBase {
/**
* Create ORC filter as a SearchArgument instance.
+ *
+ * @param schema the schema of the ORC file.
+ * @param readSchema the schema that the ORC file is read with, if it can differ from `schema`.
+ * A predicate with a value on a field of a Types Framework type is converted
+ * only if the field is read as a type that keeps the stored values, see
+ * `OrcTypeOps.acceptsReadType`.
*/
- def createFilter(schema: StructType, filters: Seq[Filter]): Option[SearchArgument] = {
- val dataTypeMap = OrcFilters.getSearchableTypeMap(schema, SQLConf.get.caseSensitiveAnalysis)
+ def createFilter(
+ schema: StructType,
+ filters: Seq[Filter],
+ readSchema: Option[StructType] = None): Option[SearchArgument] = {
+ val caseSensitive = SQLConf.get.caseSensitiveAnalysis
+ val dataTypeMap = OrcFilters.getSearchableTypeMap(schema, caseSensitive)
+ val readTypeMap = readSchema.map(OrcFilters.getSearchableTypeMap(_, caseSensitive))
+ // ORC evaluates a pushed down predicate against the stored values, which the reader can
+ // change for the read type, e.g. truncate them to a lower precision.
+ def acceptsValue(name: String): Boolean = readTypeMap.forall { readTypes =>
+ OrcTypeOps(dataTypeMap(name).fieldType).forall { ops =>
+ readTypes.get(name).exists(f => ops.acceptsReadType(f.fieldType))
+ }
+ }
// Combines all convertible filters using `And` to produce a single conjunction
- val conjunctionOptional = buildTree(convertibleFilters(dataTypeMap, filters))
+ val conjunctionOptional = buildTree(convertibleFilters(dataTypeMap, filters, acceptsValue))
conjunctionOptional.map { conjunction =>
// Then tries to build a single ORC `SearchArgument` for the conjunction predicate.
// The input predicate is fully convertible. There should not be any empty result in the
@@ -81,9 +99,14 @@ private[sql] object OrcFilters extends OrcFiltersBase {
}
}
+ /**
+ * Returns the convertible parts of `filters`. `acceptsValue` tells whether a predicate with a
+ * value can be pushed down for a field, see `createFilter`.
+ */
def convertibleFilters(
dataTypeMap: Map[String, OrcPrimitiveField],
- filters: Seq[Filter]): Seq[Filter] = {
+ filters: Seq[Filter],
+ acceptsValue: String => Boolean = _ => true): Seq[Filter] = {
import org.apache.spark.sql.sources._
def convertibleFiltersHelper(
@@ -130,7 +153,8 @@ private[sql] object OrcFilters extends OrcFiltersBase {
val childResultOptional = convertibleFiltersHelper(pred, canPartialPushDown = false)
childResultOptional.map(Not)
case other =>
- for (_ <- buildLeafSearchArgument(dataTypeMap, other, newBuilder())) yield other
+ for (_ <- buildLeafSearchArgument(dataTypeMap, other, newBuilder(), acceptsValue))
+ yield other
}
filters.flatMap { filter =>
convertibleFiltersHelper(filter, true)
@@ -227,47 +251,53 @@ private[sql] object OrcFilters extends OrcFiltersBase {
* @param dataTypeMap a map from the attribute name to its data type.
* @param expression the input filter predicates.
* @param builder the input SearchArgument.Builder.
+ * @param acceptsValue whether a predicate with a value can be pushed down for an attribute.
* @return the builder so far.
*/
private def buildLeafSearchArgument(
dataTypeMap: Map[String, OrcPrimitiveField],
expression: Filter,
- builder: Builder): Option[Builder] = {
+ builder: Builder,
+ acceptsValue: String => Boolean = _ => true): Option[Builder] = {
def getType(attribute: String): PredicateLeaf.Type =
getPredicateLeafType(dataTypeMap(attribute).fieldType)
+ // IsNull and IsNotNull have no value, so they don't depend on `acceptsValue`.
+ def canPushValue(attribute: String): Boolean =
+ dataTypeMap.contains(attribute) && acceptsValue(attribute)
+
import org.apache.spark.sql.sources._
// NOTE: For all case branches dealing with leaf predicates below, the additional `startAnd()`
// call is mandatory. ORC `SearchArgument` builder requires that all leaf predicates must be
// wrapped by a "parent" predicate (`And`, `Or`, or `Not`).
expression match {
- case EqualTo(name, value) if dataTypeMap.contains(name) =>
+ case EqualTo(name, value) if canPushValue(name) =>
val castedValue = castLiteralValue(value, dataTypeMap(name).fieldType)
Some(builder.startAnd()
.equals(dataTypeMap(name).fieldName, getType(name), castedValue).end())
- case EqualNullSafe(name, value) if dataTypeMap.contains(name) =>
+ case EqualNullSafe(name, value) if canPushValue(name) =>
val castedValue = castLiteralValue(value, dataTypeMap(name).fieldType)
Some(builder.startAnd()
.nullSafeEquals(dataTypeMap(name).fieldName, getType(name), castedValue).end())
- case LessThan(name, value) if dataTypeMap.contains(name) =>
+ case LessThan(name, value) if canPushValue(name) =>
val castedValue = castLiteralValue(value, dataTypeMap(name).fieldType)
Some(builder.startAnd()
.lessThan(dataTypeMap(name).fieldName, getType(name), castedValue).end())
- case LessThanOrEqual(name, value) if dataTypeMap.contains(name) =>
+ case LessThanOrEqual(name, value) if canPushValue(name) =>
val castedValue = castLiteralValue(value, dataTypeMap(name).fieldType)
Some(builder.startAnd()
.lessThanEquals(dataTypeMap(name).fieldName, getType(name), castedValue).end())
- case GreaterThan(name, value) if dataTypeMap.contains(name) =>
+ case GreaterThan(name, value) if canPushValue(name) =>
val castedValue = castLiteralValue(value, dataTypeMap(name).fieldType)
Some(builder.startNot()
.lessThanEquals(dataTypeMap(name).fieldName, getType(name), castedValue).end())
- case GreaterThanOrEqual(name, value) if dataTypeMap.contains(name) =>
+ case GreaterThanOrEqual(name, value) if canPushValue(name) =>
val castedValue = castLiteralValue(value, dataTypeMap(name).fieldType)
Some(builder.startNot()
.lessThan(dataTypeMap(name).fieldName, getType(name), castedValue).end())
@@ -280,7 +310,7 @@ private[sql] object OrcFilters extends OrcFiltersBase {
Some(builder.startNot()
.isNull(dataTypeMap(name).fieldName, getType(name)).end())
- case In(name, values) if dataTypeMap.contains(name) =>
+ case In(name, values) if canPushValue(name) =>
val castedValues = values.map(v => castLiteralValue(v, dataTypeMap(name).fieldType))
Some(builder.startAnd().in(dataTypeMap(name).fieldName, getType(name),
castedValues.map(_.asInstanceOf[AnyRef]): _*).end())
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/types/ops/OrcTypeOps.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/types/ops/OrcTypeOps.scala
index c0bcb1ddeabb..5eafd0f41ab2 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/types/ops/OrcTypeOps.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/types/ops/OrcTypeOps.scala
@@ -41,7 +41,8 @@ import org.apache.spark.sql.types.{DataType, TimestampLTZNanosType, TimestampNTZ
* (OrcSerializer.newConverter)
* - Row-based read (deserialize): ORC WritableComparable -> Catalyst value
* (OrcDeserializer.newWriter)
- * - Predicate pushdown: PredicateLeaf.Type mapping + literal casting (OrcFilters)
+ * - Predicate pushdown: PredicateLeaf.Type mapping + literal casting + read type check
+ * (OrcFilters)
*
* DELIBERATELY NOT ON THE TRAIT:
* - Vectorized read. ORC's vectorized path (OrcAtomicColumnVector, a Java class) dispatches via
@@ -156,6 +157,15 @@ private[orc] trait OrcTypeOps extends Serializable {
* Only meaningful for types that also return a predicateLeafType. Default is identity.
*/
def castFilterLiteral(value: Any): Any = value
+
+ /**
+ * Whether a predicate with a value is pushable for a field of this type in the ORC file that is
+ * read as `readType`, the Spark type in the read schema (OrcFilters.createFilter). ORC evaluates
+ * a pushed down predicate against the stored values, so this is false when the reader changes
+ * them for `readType`, e.g. truncates them to a lower precision. IsNull and IsNotNull are pushed
+ * down regardless. Default is true.
+ */
+ def acceptsReadType(readType: DataType): Boolean = true
}
/**
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/types/ops/TimestampNanosOrcOps.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/types/ops/TimestampNanosOrcOps.scala
index 7ea09e284822..eb17c5863233 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/types/ops/TimestampNanosOrcOps.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/types/ops/TimestampNanosOrcOps.scala
@@ -27,7 +27,7 @@ import org.apache.orc.mapred.OrcTimestamp
import org.apache.spark.sql.catalyst.expressions.SpecializedGetters
import org.apache.spark.sql.catalyst.util.DateTimeUtils
-import org.apache.spark.sql.types.{TimestampLTZNanosType, TimestampNTZNanosType}
+import org.apache.spark.sql.types.{DataType, TimestampLTZNanosType, TimestampNTZNanosType}
import org.apache.spark.unsafe.types.TimestampNanosVal
/**
@@ -92,6 +92,17 @@ case class TimestampLTZNanosOrcOps(t: TimestampLTZNanosType) extends OrcTypeOps
case i: Instant => Timestamp.valueOf(LocalDateTime.ofInstant(i, ZoneOffset.UTC))
case other => other
}
+
+ // The reader truncates the stored values to the read precision (see makeDeserializer and
+ // OrcAtomicColumnVector.getTimestampLTZNanos), while ORC evaluates a pushed down predicate
+ // against the stored values. E.g. a stored 00:00:00.123456789 of a TIMESTAMP_LTZ(9) file is read
+ // as 00:00:00.1234567 for TIMESTAMP_LTZ(7), so a pushed down `ts = 00:00:00.1234567` would skip
+ // it. The values of a TIMESTAMP_LTZ(p) file have at most p fractional digits, so a read
+ // precision of at least p keeps them. Any other read type does not keep them either.
+ override def acceptsReadType(readType: DataType): Boolean = readType match {
+ case r: TimestampLTZNanosType => r.precision >= t.precision
+ case _ => false
+ }
}
/**
@@ -154,4 +165,15 @@ case class TimestampNTZNanosOrcOps(t: TimestampNTZNanosType) extends OrcTypeOps
case ldt: LocalDateTime => Timestamp.valueOf(ldt)
case other => other
}
+
+ // The reader truncates the stored values to the read precision (see makeDeserializer and
+ // OrcAtomicColumnVector.getTimestampNTZNanos), while ORC evaluates a pushed down predicate
+ // against the stored values. E.g. a stored 00:00:00.123456789 of a TIMESTAMP_NTZ(9) file is read
+ // as 00:00:00.1234567 for TIMESTAMP_NTZ(7), so a pushed down `ts = 00:00:00.1234567` would skip
+ // it. The values of a TIMESTAMP_NTZ(p) file have at most p fractional digits, so a read
+ // precision of at least p keeps them. Any other read type does not keep them either.
+ override def acceptsReadType(readType: DataType): Boolean = readType match {
+ case r: TimestampNTZNanosType => r.precision >= t.precision
+ case _ => false
+ }
}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/orc/OrcPartitionReaderFactory.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/orc/OrcPartitionReaderFactory.scala
index 6b75b1473bd2..9732f1cf6b77 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/orc/OrcPartitionReaderFactory.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/orc/OrcPartitionReaderFactory.scala
@@ -75,7 +75,8 @@ case class OrcPartitionReaderFactory(
private def pushDownPredicates(orcSchema: TypeDescription, conf: Configuration): Unit = {
if (orcFilterPushDown && filters.nonEmpty) {
val fileSchema = OrcUtils.toCatalystSchema(orcSchema)
- OrcFilters.createFilter(fileSchema, filters.toImmutableArraySeq).foreach { f =>
+ OrcFilters.createFilter(
+ fileSchema, filters.toImmutableArraySeq, Some(readDataSchema)).foreach { f =>
OrcInputFormat.setSearchArgument(conf, f, fileSchema.fieldNames)
}
}
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilterSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilterSuite.scala
index 6481921580e7..3a48b77f3334 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilterSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilterSuite.scala
@@ -20,7 +20,7 @@ package org.apache.spark.sql.execution.datasources.orc
import java.math.MathContext
import java.nio.charset.StandardCharsets
import java.sql.{Date, Timestamp}
-import java.time.{Duration, LocalDateTime, LocalTime, Period, ZoneOffset}
+import java.time.{Duration, Instant, LocalDateTime, LocalTime, Period, ZoneOffset}
import scala.jdk.CollectionConverters._
@@ -460,6 +460,85 @@ class OrcFilterSuite extends OrcTest with SharedSparkSession {
}
}
+ test("SPARK-60397: filter pushdown - nanosecond timestamp read with a lower precision") {
+ // A TIMESTAMP(9) value 1970-01-01 00:00:00.123456789 read as TIMESTAMP(7) is truncated to
+ // 00:00:00.1234567, while ORC evaluates a pushed down predicate against the stored value.
+ // For example, a pushed down `ts = 1970-01-01 00:00:00.1234567` would skip the stripe that
+ // holds the matching row.
+ withSQLConf(
+ SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true",
+ SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
+ Seq(
+ "TIMESTAMP_LTZ" -> ((s: String) => Instant.parse(s"1970-01-01T${s}Z")),
+ "TIMESTAMP_NTZ" -> ((s: String) => LocalDateTime.parse(s"1970-01-01T$s"))
+ ).foreach { case (tsType, toValue) =>
+ withTempPath { dir =>
+ sql(s"SELECT CAST(c AS $tsType(9)) AS ts FROM VALUES " +
+ "('1970-01-01 00:00:00.123456789'), ('1970-01-01 00:00:01'), (NULL) AS t(c)")
+ .coalesce(1).write.orc(dir.getCanonicalPath)
+ Seq(7, 8).foreach { p =>
+ val readDF = spark.read.schema(s"ts $tsType($p)").orc(dir.getCanonicalPath)
+ // The stored 00:00:00.123456789 is read as `v`, e.g. 00:00:00.1234567 for p = 7.
+ val v = "00:00:00." + "123456789".take(p)
+ def lit(s: String): String = s"CAST('1970-01-01 $s' AS $tsType($p))"
+ def check(condition: String, expected: Seq[String]): Unit = {
+ checkAnswer(readDF.where(condition), expected.map(s => Row(toValue(s))))
+ }
+ withAllNativeOrcReaders {
+ check(s"ts = ${lit(v)}", Seq(v))
+ check(s"ts <=> ${lit(v)}", Seq(v))
+ check(s"ts != ${lit(v)}", Seq("00:00:01"))
+ check(s"ts < ${lit("00:00:01")}", Seq(v))
+ check(s"ts <= ${lit(v)}", Seq(v))
+ check(s"ts > ${lit(v)}", Seq("00:00:01"))
+ check(s"ts >= ${lit(v)}", Seq(v, "00:00:01"))
+ check(s"ts IN (${lit(v)}, ${lit("00:00:02")})", Seq(v))
+ checkAnswer(readDF.where("ts IS NULL"), Row(null))
+ check("ts IS NOT NULL", Seq(v, "00:00:01"))
+ }
+ }
+ }
+ }
+ }
+ }
+
+ test("SPARK-60397: nanosecond timestamp predicates with a value are pushed down only if " +
+ "the read precision is not lower than the file precision") {
+ import org.apache.spark.sql.sources._
+ // The values of a TIMESTAMP(p) file have at most p fractional digits, and the reader truncates
+ // them to the read precision. So only IsNull and IsNotNull are pushed down for a lower one.
+ def schema(precision: Int): StructType = new StructType()
+ .add("ltz", TimestampLTZNanosType(precision))
+ .add("ntz", TimestampNTZNanosType(precision))
+ val ltz = Instant.parse("1970-01-01T00:00:00.1234567Z")
+ val ntz = LocalDateTime.parse("1970-01-01T00:00:00.1234567")
+ foreachNanosPrecision { filePrecision =>
+ foreachNanosPrecision { readPrecision =>
+ def createFilter(filter: Filter): Option[SearchArgument] = OrcFilters.createFilter(
+ schema(filePrecision), Seq(filter), Some(schema(readPrecision)))
+ val clue = s"file precision $filePrecision, read precision $readPrecision"
+ Seq[(String, Any)]("ltz" -> ltz, "ntz" -> ntz).foreach { case (name, value) =>
+ Seq(IsNull(name), IsNotNull(name)).foreach { filter =>
+ assert(createFilter(filter).isDefined, s"$filter, $clue")
+ }
+ Seq(
+ EqualTo(name, value),
+ EqualNullSafe(name, value),
+ Not(EqualTo(name, value)),
+ LessThan(name, value),
+ LessThanOrEqual(name, value),
+ GreaterThan(name, value),
+ GreaterThanOrEqual(name, value),
+ In(name, Array[Any](value))
+ ).foreach { filter =>
+ assert(createFilter(filter).isDefined === (readPrecision >= filePrecision),
+ s"$filter, $clue")
+ }
+ }
+ }
+ }
+ }
+
test("filter pushdown - combinations with logical operators") {
withOrcDataFrame((1 to 4).map(i => Tuple1(Option(i)))) { implicit df =>
checkFilterPredicate(
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org