You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by su...@apache.org on 2026/09/04 16:54:14 UTC
(datafusion-comet) branch main updated: fix: prevent silent overflow when reading Parquet TIMESTAMP_MILLIS values (#5177)
This is an automated email from the ASF dual-hosted git repository.
sunchao pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new 2da32915c8 fix: prevent silent overflow when reading Parquet TIMESTAMP_MILLIS values (#5177)
2da32915c8 is described below
commit 2da32915c87909d7dbf8d62adb02f8349c36bca0
Author: Peter Lee <pe...@gmail.com>
AuthorDate: Sat Sep 5 00:54:10 2026 +0800
fix: prevent silent overflow when reading Parquet TIMESTAMP_MILLIS values (#5177)
* refactor: use Arrow casts for temporal conversions
* fix: preserve Spark temporal cast semantics
* test: match Spark micros-to-millis cases
* test: link Spark micros-to-millis cases
* test: use Spark tag in source link
* Use imported arity kernel for Spark timestamp downscaling
* fix: enforce Spark temporal conversion semantics
* andy's 3rd review
* address review: drop temporal.rs refactor, improve adapter error message
- Revert cast_date_to_timestamp to main's version to avoid conflicting
with #5457, which rewrites the same function as a safety fix
- Document the Int32 input guarantee and zero-copy reinterpret path in
date_from_unix_date
- Make the microsecond-physical/millisecond-target rejection message
actionable (report link + scan fallback workaround)
Co-Authored-By: Claude Fable 5 <no...@anthropic.com>
* chore: remove unused imports in parquet_support
Co-Authored-By: Claude Fable 5 <no...@anthropic.com>
* test: exercise dictionary-encoded pages in TIMESTAMP_MILLIS overflow regression
A single row falls back to PLAIN even with dictionary encoding enabled,
so the dictionary leg never covered a dictionary read. Write 16 repeated
rows and assert hasDictionaryEncodedPages matches the writer setting.
Co-Authored-By: Claude Fable 5 <no...@anthropic.com>
* fix: preserve timestamp pruning by rewriting predicates to the millisecond domain
The predicate over a TIMESTAMP_MILLIS file column was wrapped in
CometCastColumnExpr, which DataFusion's pruning analyzer cannot see
through, so row groups Spark prunes from millisecond statistics were
read and hit the checked millis->micros conversion. Rewrite predicate
comparisons into the millisecond domain (exact integer rescaling of the
literal, mirroring Spark's ParquetFilters) and unwrap IS NULL checks, so
pruning works and predicate evaluation never converts file values. The
scan output conversion stays checked.
Co-Authored-By: Claude Fable 5 <no...@anthropic.com>
* fix: cover IN, null-safe equality, and nested predicates in the millis-domain rewrite
Extend the millisecond-domain predicate rewrite to InListExpr and
IsDistinctFrom/IsNotDistinctFrom, both of which DataFusion's pruning
predicate can analyze, so row-group pruning protects flat TIMESTAMP_MILLIS
columns for those forms too.
Nested-field predicates can be neither pruned (PruningPredicate has no
nested-field support) nor evaluated as Parquet row filters (struct columns
are classified non-pushable), so no rewrite can keep the checked conversion
from failing queries Spark answers via nested statistics pruning. Scans
whose data filters reference nested fields fall back to the safe cast
(overflow -> NULL, the pre-existing behavior), and the checked conversion
is scoped to top-level columns for the same reason.
Co-Authored-By: Claude Fable 5 <no...@anthropic.com>
* test: cover empty NOT IN with row filters
* fix: retain dropped parquet filters
* fix: retain scalar subquery filters
---------
Co-authored-by: Claude Fable 5 <no...@anthropic.com>
---
native/core/src/execution/planner.rs | 6 +-
native/core/src/parquet/cast_column.rs | 306 +++++++--------------
native/core/src/parquet/parquet_exec.rs | 11 +-
native/core/src/parquet/parquet_support.rs | 122 +++++++-
native/core/src/parquet/schema_adapter.rs | 8 +-
native/proto/src/proto/operator.proto | 3 +
.../src/datetime_funcs/date_from_unix_date.rs | 40 +--
native/spark-expr/src/utils.rs | 9 +-
.../comet/serde/operator/CometNativeScan.scala | 4 +-
.../spark/sql/comet/CometNativeScanExec.scala | 1 +
.../apache/comet/parquet/ParquetReadSuite.scala | 216 +++++++++++++++
11 files changed, 469 insertions(+), 257 deletions(-)
diff --git a/native/core/src/execution/planner.rs b/native/core/src/execution/planner.rs
index c3ae63372f..739c4ab2f3 100644
--- a/native/core/src/execution/planner.rs
+++ b/native/core/src/execution/planner.rs
@@ -1724,7 +1724,11 @@ impl PhysicalPlanner {
object_store_url,
file_groups,
Some(projection_vector),
- Some(data_filters?),
+ if common.has_data_filters || !common.data_filters.is_empty() {
+ Some(data_filters?)
+ } else {
+ None
+ },
default_values,
common.session_timezone.as_str(),
common.case_sensitive,
diff --git a/native/core/src/parquet/cast_column.rs b/native/core/src/parquet/cast_column.rs
index 1cc928d1d5..231f88c507 100644
--- a/native/core/src/parquet/cast_column.rs
+++ b/native/core/src/parquet/cast_column.rs
@@ -15,10 +15,7 @@
// specific language governing permissions and limitations
// under the License.
use arrow::{
- array::{
- make_array, Array, ArrayRef, LargeListArray, ListArray, MapArray, StructArray,
- TimestampMicrosecondArray, TimestampMillisecondArray,
- },
+ array::{make_array, Array, ArrayRef, LargeListArray, ListArray, MapArray, StructArray},
compute::CastOptions,
datatypes::{DataType, FieldRef, Schema, TimeUnit},
record_batch::RecordBatch,
@@ -26,8 +23,7 @@ use arrow::{
use crate::parquet::parquet_support::{spark_parquet_convert, SparkParquetOptions};
use datafusion::common::format::DEFAULT_CAST_OPTIONS;
-use datafusion::common::Result as DataFusionResult;
-use datafusion::common::ScalarValue;
+use datafusion::common::{DataFusionError, Result as DataFusionResult};
use datafusion::logical_expr::ColumnarValue;
use datafusion::physical_expr::PhysicalExpr;
use std::{
@@ -142,40 +138,6 @@ fn relabel_array(array: ArrayRef, target_type: &DataType) -> ArrayRef {
}
}
-/// Casts a Timestamp(Microsecond) array to Timestamp(Millisecond) by dividing values by 1000.
-/// Preserves the timezone from the target type.
-fn cast_timestamp_micros_to_millis_array(
- array: &ArrayRef,
- target_tz: Option<Arc<str>>,
-) -> ArrayRef {
- let micros_array = array
- .as_any()
- .downcast_ref::<TimestampMicrosecondArray>()
- .expect("Expected TimestampMicrosecondArray");
-
- let millis_values: TimestampMillisecondArray =
- arrow::compute::kernels::arity::unary(micros_array, |v| v / 1000);
-
- // Apply timezone if present
- let result = if let Some(tz) = target_tz {
- millis_values.with_timezone(tz)
- } else {
- millis_values
- };
-
- Arc::new(result)
-}
-
-/// Casts a Timestamp(Microsecond) scalar to Timestamp(Millisecond) by dividing the value by 1000.
-/// Preserves the timezone from the target type.
-fn cast_timestamp_micros_to_millis_scalar(
- opt_val: Option<i64>,
- target_tz: Option<Arc<str>>,
-) -> ScalarValue {
- let new_val = opt_val.map(|v| v / 1000);
- ScalarValue::TimestampMillisecond(new_val, target_tz)
-}
-
#[derive(Debug, Clone, Eq)]
pub struct CometCastColumnExpr {
/// The physical expression producing the value to cast.
@@ -214,20 +176,41 @@ impl Hash for CometCastColumnExpr {
}
impl CometCastColumnExpr {
- /// Create a new [`CometCastColumnExpr`].
- pub fn new(
+ /// Try to create a new [`CometCastColumnExpr`].
+ pub fn try_new(
expr: Arc<dyn PhysicalExpr>,
physical_field: FieldRef,
target_field: FieldRef,
cast_options: Option<CastOptions<'static>>,
- ) -> Self {
- Self {
+ ) -> DataFusionResult<Self> {
+ let physical_type = physical_field.data_type();
+ let target_type = target_field.data_type();
+ // `target_field` is the Spark logical field, while `physical_field` comes from the
+ // Parquet or Iceberg file. Comet represents Spark's TimestampType and TimestampNTZType
+ // as Arrow microseconds, and Spark maps both TIMESTAMP_MICROS and TIMESTAMP_MILLIS files
+ // to those logical types. For a top-level timestamp column, a millisecond target is
+ // therefore invalid at this read-adapter boundary:
+ // https://github.com/apache/spark/blob/v4.2.0/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L318-L324
+ if matches!(
+ (physical_type, target_type),
+ (
+ DataType::Timestamp(TimeUnit::Microsecond, _),
+ DataType::Timestamp(TimeUnit::Millisecond, _)
+ )
+ ) {
+ return Err(DataFusionError::Plan(format!(
+ "Cannot adapt Spark timestamp field '{}' from {physical_type} to {target_type}: Spark read schemas represent logical timestamps in microseconds. This indicates a bug in Comet's schema handling; please report it at https://github.com/apache/datafusion-comet/issues. As a workaround, set spark.comet.scan.enabled=false",
+ physical_field.name()
+ )));
+ }
+
+ Ok(Self {
expr,
input_physical_field: physical_field,
target_field,
cast_options: cast_options.unwrap_or(DEFAULT_CAST_OPTIONS),
parquet_options: None,
- }
+ })
}
/// Set Spark parquet options to enable complex nested type conversions.
@@ -271,23 +254,7 @@ impl PhysicalExpr for CometCastColumnExpr {
let input_physical_field = self.input_physical_field.data_type();
let target_field = self.target_field.data_type();
- // Handle specific type conversions with custom casts
match (input_physical_field, target_field) {
- // Timestamp(Microsecond) -> Timestamp(Millisecond)
- (
- DataType::Timestamp(TimeUnit::Microsecond, _),
- DataType::Timestamp(TimeUnit::Millisecond, target_tz),
- ) => match value {
- ColumnarValue::Array(array) => {
- let casted = cast_timestamp_micros_to_millis_array(&array, target_tz.clone());
- Ok(ColumnarValue::Array(casted))
- }
- ColumnarValue::Scalar(ScalarValue::TimestampMicrosecond(opt_val, _)) => {
- let casted = cast_timestamp_micros_to_millis_scalar(opt_val, target_tz.clone());
- Ok(ColumnarValue::Scalar(casted))
- }
- _ => Ok(value),
- },
// Nested types that differ only in field names (e.g., List element named
// "item" vs "element", or Map entries named "key_value" vs "entries").
// Re-label the array so the DataType metadata matches the logical schema.
@@ -329,12 +296,12 @@ impl PhysicalExpr for CometCastColumnExpr {
) -> DataFusionResult<Arc<dyn PhysicalExpr>> {
assert_eq!(children.len(), 1);
let child = children.pop().expect("CastColumnExpr child");
- let mut new_expr = Self::new(
+ let mut new_expr = Self::try_new(
child,
Arc::clone(&self.input_physical_field),
Arc::clone(&self.target_field),
Some(self.cast_options.clone()),
- );
+ )?;
if let Some(opts) = &self.parquet_options {
new_expr = new_expr.with_parquet_options(opts.clone());
}
@@ -349,159 +316,84 @@ impl PhysicalExpr for CometCastColumnExpr {
#[cfg(test)]
mod tests {
use super::*;
- use arrow::array::{Array, Int32Array, StringArray};
+ use arrow::array::{
+ Array, Int32Array, StringArray, TimestampMicrosecondArray, TimestampMillisecondArray,
+ };
use arrow::datatypes::{Field, Fields};
use datafusion::physical_expr::expressions::Column;
+ use datafusion_comet_spark_expr::EvalMode;
#[test]
- fn test_cast_timestamp_micros_to_millis_array() {
- // Create a TimestampMicrosecond array with some values
- let micros_array: TimestampMicrosecondArray = vec![
- Some(1_000_000), // 1 second in micros
- Some(2_500_000), // 2.5 seconds in micros
- None, // null value
- Some(0), // zero
- Some(-1_000_000), // negative value (before epoch)
- ]
- .into();
- let array_ref: ArrayRef = Arc::new(micros_array);
-
- // Cast without timezone
- let result = cast_timestamp_micros_to_millis_array(&array_ref, None);
- let millis_array = result
- .as_any()
- .downcast_ref::<TimestampMillisecondArray>()
- .expect("Expected TimestampMillisecondArray");
-
- assert_eq!(millis_array.len(), 5);
- assert_eq!(millis_array.value(0), 1000); // 1_000_000 / 1000
- assert_eq!(millis_array.value(1), 2500); // 2_500_000 / 1000
- assert!(millis_array.is_null(2));
- assert_eq!(millis_array.value(3), 0);
- assert_eq!(millis_array.value(4), -1000); // -1_000_000 / 1000
- }
-
- #[test]
- fn test_cast_timestamp_micros_to_millis_array_with_timezone() {
- let micros_array: TimestampMicrosecondArray = vec![Some(1_000_000), Some(2_000_000)].into();
- let array_ref: ArrayRef = Arc::new(micros_array);
-
- let target_tz: Option<Arc<str>> = Some(Arc::from("UTC"));
- let result = cast_timestamp_micros_to_millis_array(&array_ref, target_tz);
- let millis_array = result
- .as_any()
- .downcast_ref::<TimestampMillisecondArray>()
- .expect("Expected TimestampMillisecondArray");
-
- assert_eq!(millis_array.value(0), 1000);
- assert_eq!(millis_array.value(1), 2000);
- // Verify timezone is preserved
- assert_eq!(
- result.data_type(),
- &DataType::Timestamp(TimeUnit::Millisecond, Some(Arc::from("UTC")))
- );
- }
-
- #[test]
- fn test_cast_timestamp_micros_to_millis_scalar() {
- // Test with a value
- let result = cast_timestamp_micros_to_millis_scalar(Some(1_500_000), None);
- assert_eq!(result, ScalarValue::TimestampMillisecond(Some(1500), None));
-
- // Test with null
- let null_result = cast_timestamp_micros_to_millis_scalar(None, None);
- assert_eq!(null_result, ScalarValue::TimestampMillisecond(None, None));
-
- // Test with timezone
- let target_tz: Option<Arc<str>> = Some(Arc::from("UTC"));
- let tz_result = cast_timestamp_micros_to_millis_scalar(Some(2_000_000), target_tz.clone());
- assert_eq!(
- tz_result,
- ScalarValue::TimestampMillisecond(Some(2000), target_tz)
- );
- }
-
- #[test]
- fn test_comet_cast_column_expr_evaluate_micros_to_millis_array() {
- // Create input schema with TimestampMicrosecond column
- let input_field = Arc::new(Field::new(
- "ts",
- DataType::Timestamp(TimeUnit::Microsecond, None),
- true,
- ));
- let schema = Schema::new(vec![Arc::clone(&input_field)]);
-
- // Create target field with TimestampMillisecond
- let target_field = Arc::new(Field::new(
- "ts",
- DataType::Timestamp(TimeUnit::Millisecond, None),
- true,
- ));
-
- // Create a column expression
- let col_expr: Arc<dyn PhysicalExpr> = Arc::new(Column::new("ts", 0));
-
- // Create the CometCastColumnExpr
- let cast_expr = CometCastColumnExpr::new(col_expr, input_field, target_field, None);
-
- // Create a record batch with TimestampMicrosecond data
- let micros_array: TimestampMicrosecondArray =
- vec![Some(1_000_000), Some(2_000_000), None].into();
- let batch = RecordBatch::try_new(Arc::new(schema), vec![Arc::new(micros_array)]).unwrap();
-
- // Evaluate
- let result = cast_expr.evaluate(&batch).unwrap();
-
- match result {
- ColumnarValue::Array(arr) => {
- let millis_array = arr
- .as_any()
- .downcast_ref::<TimestampMillisecondArray>()
- .expect("Expected TimestampMillisecondArray");
- assert_eq!(millis_array.value(0), 1000);
- assert_eq!(millis_array.value(1), 2000);
- assert!(millis_array.is_null(2));
- }
- _ => panic!("Expected Array result"),
+ fn test_rejects_millisecond_logical_timestamp() {
+ for timezone in [None, Some(Arc::from("UTC"))] {
+ let input_field = Arc::new(Field::new(
+ "ts",
+ DataType::Timestamp(TimeUnit::Microsecond, timezone.clone()),
+ true,
+ ));
+ let target_field = Arc::new(Field::new(
+ "ts",
+ DataType::Timestamp(TimeUnit::Millisecond, timezone),
+ true,
+ ));
+ let expr: Arc<dyn PhysicalExpr> = Arc::new(Column::new("ts", 0));
+
+ let err = CometCastColumnExpr::try_new(expr, input_field, target_field, None)
+ .expect_err("millisecond logical timestamp must be rejected during planning");
+ assert!(matches!(
+ err,
+ DataFusionError::Plan(message)
+ if message.contains("Spark read schemas represent logical timestamps in microseconds")
+ ));
}
}
#[test]
- fn test_comet_cast_column_expr_evaluate_micros_to_millis_scalar() {
- // Create input schema with TimestampMicrosecond column
- let input_field = Arc::new(Field::new(
- "ts",
- DataType::Timestamp(TimeUnit::Microsecond, None),
- true,
- ));
- let schema = Schema::new(vec![Arc::clone(&input_field)]);
-
- // Create target field with TimestampMillisecond
- let target_field = Arc::new(Field::new(
- "ts",
- DataType::Timestamp(TimeUnit::Millisecond, None),
- true,
- ));
-
- // Create a literal expression that returns a scalar
- let scalar = ScalarValue::TimestampMicrosecond(Some(1_500_000), None);
- let literal_expr: Arc<dyn PhysicalExpr> =
- Arc::new(datafusion::physical_expr::expressions::Literal::new(scalar));
-
- // Create the CometCastColumnExpr
- let cast_expr = CometCastColumnExpr::new(literal_expr, input_field, target_field, None);
-
- // Create an empty batch (scalar doesn't need data)
- let batch = RecordBatch::new_empty(Arc::new(schema));
-
- // Evaluate
- let result = cast_expr.evaluate(&batch).unwrap();
-
- match result {
- ColumnarValue::Scalar(s) => {
- assert_eq!(s, ScalarValue::TimestampMillisecond(Some(1500), None));
+ fn test_parquet_millis_to_micros_uses_checked_multiply() {
+ // Spark's Parquet reader calls the checked `millisToMicros` conversion for both
+ // direct and dictionary values:
+ // https://github.com/apache/spark/blob/v4.2.0/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java#L817-L833
+ for eval_mode in [EvalMode::Legacy, EvalMode::Try, EvalMode::Ansi] {
+ for (source_tz, target_tz) in [
+ (None, None),
+ (Some(Arc::from("UTC")), Some(Arc::from("UTC"))),
+ ] {
+ let input_field = Arc::new(Field::new(
+ "ts",
+ DataType::Timestamp(TimeUnit::Millisecond, source_tz.clone()),
+ true,
+ ));
+ let schema = Arc::new(Schema::new(vec![Arc::clone(&input_field)]));
+ let target_type = DataType::Timestamp(TimeUnit::Microsecond, target_tz.clone());
+ let target_field = Arc::new(Field::new("ts", target_type.clone(), true));
+ let expr: Arc<dyn PhysicalExpr> = Arc::new(Column::new("ts", 0));
+ let cast_expr = CometCastColumnExpr::try_new(expr, input_field, target_field, None)
+ .unwrap()
+ .with_parquet_options(SparkParquetOptions::new(eval_mode, "UTC", false));
+
+ let input = TimestampMillisecondArray::from(vec![Some(1_234), Some(-1_234), None])
+ .with_timezone_opt(source_tz.clone());
+ let batch =
+ RecordBatch::try_new(Arc::clone(&schema), vec![Arc::new(input)]).unwrap();
+ let ColumnarValue::Array(output) = cast_expr.evaluate(&batch).unwrap() else {
+ panic!("Expected array result");
+ };
+ let output = output
+ .as_any()
+ .downcast_ref::<TimestampMicrosecondArray>()
+ .expect("Expected TimestampMicrosecondArray");
+ assert_eq!(
+ output.iter().collect::<Vec<_>>(),
+ vec![Some(1_234_000), Some(-1_234_000), None]
+ );
+ assert_eq!(output.data_type(), &target_type);
+
+ let overflow =
+ TimestampMillisecondArray::from(vec![i64::MAX]).with_timezone_opt(source_tz);
+ let batch =
+ RecordBatch::try_new(Arc::clone(&schema), vec![Arc::new(overflow)]).unwrap();
+ assert!(cast_expr.evaluate(&batch).is_err());
}
- _ => panic!("Expected Scalar result"),
}
}
diff --git a/native/core/src/parquet/parquet_exec.rs b/native/core/src/parquet/parquet_exec.rs
index 12f152d099..f18860a9fa 100644
--- a/native/core/src/parquet/parquet_exec.rs
+++ b/native/core/src/parquet/parquet_exec.rs
@@ -55,8 +55,9 @@ use std::sync::Arc;
///
/// `projection_vector`: A vector of the indexes in the schema of the fields to be projected
///
-/// `data_filters`: Any predicate that must be applied to the data returned by the scan. If
-/// specified, then `data_schema` must also be specified.
+/// `data_filters`: Any predicate that must be applied to the data returned by the scan. An empty
+/// `Vec` means Spark supplied filters that Comet could not serialize. If specified, then
+/// `data_schema` must also be specified.
#[allow(clippy::too_many_arguments)]
pub(crate) fn init_datasource_exec(
required_schema: SchemaRef,
@@ -94,6 +95,12 @@ pub(crate) fn init_datasource_exec(
);
spark_parquet_options.use_field_id = use_field_id;
spark_parquet_options.ignore_missing_field_id = ignore_missing_field_id;
+ // Spark can discard filtered-out values before timestamp conversion using statistics,
+ // dictionary, and row-level filters. Comet cannot mirror every pruning path, so applying
+ // checked conversion in a filtered scan can fail on values Spark never reads. Preserve the
+ // existing safe cast for filtered scans and use checked conversion only when every value is
+ // necessarily read.
+ spark_parquet_options.checked_timestamp_overflow = data_filters.is_none();
// Determine the schema and projection to use for ParquetSource.
// When data_schema is provided, use it as the base schema so DataFusion knows the full
diff --git a/native/core/src/parquet/parquet_support.rs b/native/core/src/parquet/parquet_support.rs
index 638957317a..fe16e099b7 100644
--- a/native/core/src/parquet/parquet_support.rs
+++ b/native/core/src/parquet/parquet_support.rs
@@ -23,8 +23,8 @@ use arrow::compute::can_cast_types;
use arrow::datatypes::{FieldRef, Fields};
use arrow::{
array::{
- cast::AsArray, new_null_array, types::TimestampMicrosecondType, Array, ArrayRef,
- StructArray,
+ cast::AsArray, new_null_array, types::TimestampMicrosecondType,
+ types::TimestampMillisecondType, Array, ArrayRef, ArrowNativeTypeOp, StructArray,
},
compute::{cast_with_options, CastOptions},
datatypes::{DataType, TimeUnit},
@@ -101,6 +101,12 @@ pub struct SparkParquetOptions {
/// (Spark 3.x, SPARK-36182). Mirrors Comet's per-Spark-version constant
/// in ShimCometConf.
pub allow_timestamp_ltz_to_ntz: bool,
+ /// When true (the default), a top-level TIMESTAMP_MILLIS column that overflows during
+ /// the millis->micros upscale raises an error, matching Spark's checked
+ /// `millisToMicros`. Filtered scans set this to false and retain the safe cast
+ /// (overflow -> NULL), because Spark may discard values through pruning paths that
+ /// DataFusion cannot fully mirror before conversion.
+ pub checked_timestamp_overflow: bool,
}
impl SparkParquetOptions {
@@ -117,6 +123,7 @@ impl SparkParquetOptions {
ignore_missing_field_id: false,
allow_type_promotion: false,
allow_timestamp_ltz_to_ntz: false,
+ checked_timestamp_overflow: true,
}
}
@@ -133,6 +140,7 @@ impl SparkParquetOptions {
ignore_missing_field_id: false,
allow_type_promotion: false,
allow_timestamp_ltz_to_ntz: false,
+ checked_timestamp_overflow: true,
}
}
}
@@ -169,6 +177,15 @@ fn parquet_convert_array(
array: ArrayRef,
to_type: &DataType,
parquet_options: &SparkParquetOptions,
+) -> DataFusionResult<ArrayRef> {
+ parquet_convert_array_impl(array, to_type, parquet_options, true)
+}
+
+fn parquet_convert_array_impl(
+ array: ArrayRef,
+ to_type: &DataType,
+ parquet_options: &SparkParquetOptions,
+ top_level: bool,
) -> DataFusionResult<ArrayRef> {
use DataType::*;
let from_type = array.data_type();
@@ -184,10 +201,11 @@ fn parquet_convert_array(
)?),
(List(_), List(to_inner_type)) => {
let list_arr: &ListArray = array.as_list();
- let cast_field = parquet_convert_array(
+ let cast_field = parquet_convert_array_impl(
Arc::clone(list_arr.values()),
to_inner_type.data_type(),
parquet_options,
+ false,
)?;
Ok(Arc::new(ListArray::new(
@@ -197,6 +215,28 @@ fn parquet_convert_array(
list_arr.nulls().cloned(),
)))
}
+ (
+ Timestamp(TimeUnit::Millisecond, _),
+ Timestamp(TimeUnit::Microsecond, target_tz),
+ ) if top_level && parquet_options.checked_timestamp_overflow => {
+ // Spark's Parquet reader calls the checked `millisToMicros` conversion for both
+ // direct and dictionary values, independent of CAST evaluation mode:
+ // https://github.com/apache/spark/blob/v4.2.0/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java#L817-L833
+ // `millisToMicros` uses `Math.multiplyExact`:
+ // https://github.com/apache/spark/blob/v4.2.0/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala#L103-L108
+ //
+ // The checked conversion is limited to TOP-LEVEL columns. Spark only avoids the
+ // error for filtered-out values through row-group statistics pruning, and
+ // DataFusion's PruningPredicate does not support nested fields yet, so a checked
+ // conversion on a nested field would fail queries whose predicates Spark prunes
+ // (e.g. `WHERE s.ts < X` over an all-overflowing file). Nested fields keep the
+ // pre-existing safe-cast behavior below (overflow -> NULL).
+ let micros = array
+ .as_primitive::<TimestampMillisecondType>()
+ .try_unary::<_, TimestampMicrosecondType, _>(|value| value.mul_checked(1_000))?
+ .with_timezone_opt(target_tz.clone());
+ Ok(Arc::new(micros))
+ }
(Timestamp(TimeUnit::Microsecond, None), Timestamp(TimeUnit::Microsecond, Some(tz))) => {
Ok(Arc::new(
array
@@ -328,10 +368,11 @@ fn parquet_convert_struct_to_struct(
};
if let Some(from_index) = from_index {
- cast_fields.push(parquet_convert_array(
+ cast_fields.push(parquet_convert_array_impl(
Arc::clone(array.column(from_index)),
to_field.data_type(),
parquet_options,
+ false,
)?);
field_overlap = true;
} else {
@@ -379,15 +420,17 @@ fn parquet_convert_map_to_map(
"map is missing value field".to_string(),
))?;
- let key_array = parquet_convert_array(
+ let key_array = parquet_convert_array_impl(
Arc::clone(from.keys()),
key_field.data_type(),
parquet_options,
+ false,
)?;
- let value_array = parquet_convert_array(
+ let value_array = parquet_convert_array_impl(
Arc::clone(from.values()),
value_field.data_type(),
parquet_options,
+ false,
)?;
Ok(Arc::new(MapArray::new(
@@ -685,4 +728,71 @@ mod tests {
}
}
}
+
+ #[test]
+ fn test_millis_to_micros_overflow_checked_only_at_top_level() {
+ use crate::parquet::parquet_support::{parquet_convert_array, SparkParquetOptions};
+ use arrow::array::{Array, ArrayRef, StructArray, TimestampMillisecondArray};
+ use arrow::datatypes::{DataType, Field, Fields, TimeUnit};
+ use datafusion_comet_spark_expr::EvalMode;
+ use std::sync::Arc;
+
+ let options = SparkParquetOptions::new(EvalMode::Legacy, "UTC", false);
+ let overflow_millis = 9_223_372_036_854_776_i64;
+ let millis: ArrayRef = Arc::new(TimestampMillisecondArray::from(vec![
+ Some(overflow_millis),
+ None,
+ ]));
+ let micros_type = DataType::Timestamp(TimeUnit::Microsecond, None);
+
+ // Top-level: checked, matching Spark's `millisToMicros` (`Math.multiplyExact`).
+ let err = parquet_convert_array(Arc::clone(&millis), µs_type, &options)
+ .expect_err("top-level overflow must error");
+ assert!(
+ err.to_string().to_lowercase().contains("overflow"),
+ "unexpected error: {err}"
+ );
+
+ // Filtered scans disable checked conversion because Spark may prune values before
+ // conversion through paths DataFusion cannot fully mirror.
+ let mut unchecked_options = options.clone();
+ unchecked_options.checked_timestamp_overflow = false;
+ let converted =
+ parquet_convert_array(Arc::clone(&millis), µs_type, &unchecked_options)
+ .expect("unchecked overflow must not error");
+ assert!(converted.is_null(0), "overflow must become NULL");
+ assert!(converted.is_null(1));
+
+ // Nested: DataFusion's PruningPredicate cannot prune nested fields, so a
+ // checked conversion would fail queries whose predicates Spark satisfies via
+ // row-group statistics pruning. The nested field keeps the safe-cast behavior:
+ // overflow becomes NULL.
+ let child_field = Arc::new(Field::new(
+ "ts",
+ DataType::Timestamp(TimeUnit::Millisecond, None),
+ true,
+ ));
+ let strukt: ArrayRef = Arc::new(StructArray::new(
+ Fields::from(vec![Arc::clone(&child_field)]),
+ vec![millis],
+ None,
+ ));
+ let target = DataType::Struct(Fields::from(vec![Arc::new(Field::new(
+ "ts",
+ micros_type.clone(),
+ true,
+ ))]));
+ let converted = parquet_convert_array(strukt, &target, &options)
+ .expect("nested overflow must not error");
+ let converted_child = Arc::clone(
+ converted
+ .as_any()
+ .downcast_ref::<StructArray>()
+ .unwrap()
+ .column(0),
+ );
+ assert_eq!(converted_child.data_type(), µs_type);
+ assert!(converted_child.is_null(0), "overflow must become NULL");
+ assert!(converted_child.is_null(1));
+ }
}
diff --git a/native/core/src/parquet/schema_adapter.rs b/native/core/src/parquet/schema_adapter.rs
index 2722892bc6..8b36261822 100644
--- a/native/core/src/parquet/schema_adapter.rs
+++ b/native/core/src/parquet/schema_adapter.rs
@@ -667,12 +667,12 @@ impl SparkPhysicalExprAdapter {
}
let cast_expr: Arc<dyn PhysicalExpr> = Arc::new(
- CometCastColumnExpr::new(
+ CometCastColumnExpr::try_new(
remapped,
Arc::clone(physical_field),
Arc::clone(logical_field),
None,
- )
+ )?
.with_parquet_options(self.parquet_options.clone()),
);
return Ok(Transformed::yes(cast_expr));
@@ -956,12 +956,12 @@ impl SparkPhysicalExprAdapter {
| (DataType::Timestamp(_, _), DataType::Int64)
) {
let comet_cast: Arc<dyn PhysicalExpr> = Arc::new(
- CometCastColumnExpr::new(
+ CometCastColumnExpr::try_new(
child,
input_field,
Arc::clone(cast.target_field()),
None,
- )
+ )?
.with_parquet_options(self.parquet_options.clone()),
);
return Ok(Transformed::yes(comet_cast));
diff --git a/native/proto/src/proto/operator.proto b/native/proto/src/proto/operator.proto
index 01bfa70c21..7b34f84012 100644
--- a/native/proto/src/proto/operator.proto
+++ b/native/proto/src/proto/operator.proto
@@ -176,6 +176,9 @@ message NativeScanCommon {
// SchemaColumnConvertNotSupportedException (Spark 3.x, SPARK-36182). Set
// from Comet's per-Spark-version constant in ShimCometConf.
bool allow_timestamp_ltz_to_ntz = 18;
+ // True when Spark supplied a Parquet data filter, including when Comet could
+ // not serialize any of those filters into data_filters.
+ bool has_data_filters = 19;
}
message NativeScan {
diff --git a/native/spark-expr/src/datetime_funcs/date_from_unix_date.rs b/native/spark-expr/src/datetime_funcs/date_from_unix_date.rs
index 0e624e6472..2d08fd2b76 100644
--- a/native/spark-expr/src/datetime_funcs/date_from_unix_date.rs
+++ b/native/spark-expr/src/datetime_funcs/date_from_unix_date.rs
@@ -15,13 +15,12 @@
// specific language governing permissions and limitations
// under the License.
-use arrow::array::{Array, Date32Array, Int32Array};
+use arrow::compute::cast_with_options;
use arrow::datatypes::DataType;
-use datafusion::common::{utils::take_function_args, DataFusionError, Result, ScalarValue};
+use datafusion::common::{format::DEFAULT_CAST_OPTIONS, utils::take_function_args, Result};
use datafusion::logical_expr::{
ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, Volatility,
};
-use std::sync::Arc;
/// Spark-compatible date_from_unix_date function.
/// Converts an integer representing days since Unix epoch (1970-01-01) to a Date32 value.
@@ -61,33 +60,18 @@ impl ScalarUDFImpl for SparkDateFromUnixDate {
fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
let [unix_date] = take_function_args(self.name(), args.args)?;
+ // The input is guaranteed to be Int32 by `Signature::exact` and the Comet serde
+ // (Spark's DateFromUnixDate only accepts IntegerType), so the casts below take
+ // Arrow's zero-copy `Int32 -> Date32` reinterpret path.
match unix_date {
- ColumnarValue::Array(arr) => {
- let int_array = arr.as_any().downcast_ref::<Int32Array>().ok_or_else(|| {
- DataFusionError::Execution(
- "date_from_unix_date expects Int32Array input".to_string(),
- )
- })?;
-
- // Date32 and Int32 both represent days since epoch, so we can directly
- // reinterpret the values. The only operation needed is creating a Date32Array
- // from the same underlying i32 values.
- let date_array =
- Date32Array::new(int_array.values().clone(), int_array.nulls().cloned());
-
- Ok(ColumnarValue::Array(Arc::new(date_array)))
+ ColumnarValue::Array(arr) => Ok(ColumnarValue::Array(cast_with_options(
+ arr.as_ref(),
+ &DataType::Date32,
+ &DEFAULT_CAST_OPTIONS,
+ )?)),
+ ColumnarValue::Scalar(scalar) => {
+ Ok(ColumnarValue::Scalar(scalar.cast_to(&DataType::Date32)?))
}
- ColumnarValue::Scalar(scalar) => match scalar {
- ScalarValue::Int32(Some(days)) => {
- Ok(ColumnarValue::Scalar(ScalarValue::Date32(Some(days))))
- }
- ScalarValue::Int32(None) | ScalarValue::Null => {
- Ok(ColumnarValue::Scalar(ScalarValue::Date32(None)))
- }
- _ => Err(DataFusionError::Execution(
- "date_from_unix_date expects Int32 scalar input".to_string(),
- )),
- },
}
}
diff --git a/native/spark-expr/src/utils.rs b/native/spark-expr/src/utils.rs
index 02a14545d6..7ab665c195 100644
--- a/native/spark-expr/src/utils.rs
+++ b/native/spark-expr/src/utils.rs
@@ -29,7 +29,6 @@ use std::sync::Arc;
use arrow::array::timezone::Tz;
use arrow::array::types::TimestampMillisecondType;
-use arrow::array::TimestampMicrosecondArray;
use arrow::datatypes::{MAX_DECIMAL128_FOR_EACH_PRECISION, MIN_DECIMAL128_FOR_EACH_PRECISION};
use arrow::error::ArrowError;
use arrow::{
@@ -81,13 +80,6 @@ pub fn array_with_timezone(
// so the result has the exact annotation the caller expects.
timestamp_ntz_to_timestamp(array, timezone.as_str(), Some(target_tz.as_ref()))
}
- Some(DataType::Timestamp(TimeUnit::Microsecond, None)) => {
- // Convert from Timestamp(Millisecond, None) to Timestamp(Microsecond, None)
- let millis_array = as_primitive_array::<TimestampMillisecondType>(&array);
- let micros_array: TimestampMicrosecondArray =
- arrow::compute::kernels::arity::unary(millis_array, |v| v * 1000);
- Ok(Arc::new(micros_array))
- }
_ => {
// Not supported
Err(ArrowError::CastError(format!(
@@ -376,6 +368,7 @@ pub fn unlikely(b: bool) -> bool {
#[cfg(test)]
mod tests {
use super::*;
+ use arrow::array::TimestampMicrosecondArray;
fn array_containing(local_datetime: &str) -> ArrayRef {
let dt = NaiveDateTime::parse_from_str(local_datetime, "%Y-%m-%d %H:%M:%S").unwrap();
diff --git a/spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala b/spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala
index 7e6217d1c8..52c6959cdc 100644
--- a/spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala
+++ b/spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala
@@ -155,8 +155,10 @@ object CometNativeScan extends CometOperatorSerde[CometScanExec] with CometTypeS
builder.clearChildren()
if (scan.conf.getConf(SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED)) {
+ val supportedDataFilters = scan.supportedDataFilters
+ commonBuilder.setHasDataFilters(supportedDataFilters.nonEmpty)
val dataFilters = new ListBuffer[Expr]()
- for (filter <- scan.supportedDataFilters) {
+ for (filter <- supportedDataFilters) {
exprToProto(filter, scan.output) match {
case Some(proto) => dataFilters += proto
case _ =>
diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/CometNativeScanExec.scala b/spark/src/main/scala/org/apache/spark/sql/comet/CometNativeScanExec.scala
index fbebc42004..303e8279c3 100644
--- a/spark/src/main/scala/org/apache/spark/sql/comet/CometNativeScanExec.scala
+++ b/spark/src/main/scala/org/apache/spark/sql/comet/CometNativeScanExec.scala
@@ -200,6 +200,7 @@ case class CometNativeScanExec(
})
if (resolvedFilters.nonEmpty) {
val commonBuilder = base.toBuilder
+ commonBuilder.setHasDataFilters(true)
for (filter <- resolvedFilters) {
exprToProto(filter, output) match {
case Some(proto) => commonBuilder.addDataFilters(proto)
diff --git a/spark/src/test/scala/org/apache/comet/parquet/ParquetReadSuite.scala b/spark/src/test/scala/org/apache/comet/parquet/ParquetReadSuite.scala
index 8553ef2329..799197bf33 100644
--- a/spark/src/test/scala/org/apache/comet/parquet/ParquetReadSuite.scala
+++ b/spark/src/test/scala/org/apache/comet/parquet/ParquetReadSuite.scala
@@ -38,6 +38,7 @@ import org.apache.parquet.io.api.Binary
import org.apache.parquet.schema.MessageTypeParser
import org.apache.spark.SparkException
import org.apache.spark.sql.{CometTestBase, DataFrame, Row}
+import org.apache.spark.sql.catalyst.optimizer.{ConvertToLocalRelation, OptimizeIn}
import org.apache.spark.sql.catalyst.util.DateTimeUtils
import org.apache.spark.sql.comet.{CometNativeScanExec, CometScanExec}
import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
@@ -48,6 +49,7 @@ import org.apache.spark.sql.types._
import com.google.common.primitives.UnsignedLong
import org.apache.comet.CometConf
+import org.apache.comet.CometSparkSessionExtensions.isSpark40Plus
import org.apache.comet.vector.CometVector
abstract class ParquetReadSuite extends CometTestBase {
@@ -262,6 +264,220 @@ abstract class ParquetReadSuite extends CometTestBase {
}
}
+ test("TIMESTAMP_MILLIS overflow fails in native scan") {
+ // Spark routes both TimestampType and TimestampNTZType through LongAsMicrosUpdater:
+ // https://github.com/apache/spark/blob/v4.2.0/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java#L140-L164
+ // The updater calls checked millisToMicros for direct and dictionary values:
+ // https://github.com/apache/spark/blob/v4.2.0/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java#L800-L833
+ // Matches Spark's positive and negative overflow cases:
+ // https://github.com/apache/spark/blob/v4.2.0/sql/core/src/test/resources/sql-tests/inputs/timestamp.sql#L74-L83
+ def isOverflow(error: Throwable): Boolean =
+ Iterator
+ .iterate(error)(_.getCause)
+ .takeWhile(_ != null)
+ .exists(cause => Option(cause.getMessage).exists(_.toLowerCase.contains("overflow")))
+
+ Seq(false, true).foreach { dictionaryEnabled =>
+ Seq(92233720368547758L, -92233720368547758L).foreach { millis =>
+ withTempDir { dir =>
+ val path = new Path(dir.toURI.toString, "part-r-0.parquet")
+ val schema = MessageTypeParser.parseMessageType("""
+ |message root {
+ | optional int64 ts(TIMESTAMP_MILLIS);
+ | optional int64 ts_ntz(TIMESTAMP(MILLIS,false));
+ |}
+ |""".stripMargin)
+ val writer = createParquetWriter(schema, path, dictionaryEnabled)
+ // A single row falls back to PLAIN even with dictionary encoding enabled, because
+ // a one-entry dictionary page is not smaller than the raw values. Write enough
+ // repeated rows for the writer to keep dictionary-encoded data pages.
+ (0 until 16).foreach { _ =>
+ val record = new SimpleGroup(schema)
+ record.add(0, millis)
+ record.add(1, millis)
+ writer.write(record)
+ }
+ writer.close()
+
+ val footerReader = org.apache.parquet.hadoop.ParquetFileReader.open(
+ org.apache.parquet.hadoop.util.HadoopInputFile
+ .fromPath(path, spark.sessionState.newHadoopConf()))
+ try {
+ footerReader.getFooter.getBlocks.forEach { block =>
+ block.getColumns.forEach { column =>
+ assert(
+ column.getEncodingStats.hasDictionaryEncodedPages == dictionaryEnabled,
+ s"expected hasDictionaryEncodedPages=$dictionaryEnabled " +
+ s"for column ${column.getPath}")
+ }
+ }
+ } finally {
+ footerReader.close()
+ }
+
+ Seq(false, true).foreach { ansiEnabled =>
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+ readParquetFile(path.toString) { df =>
+ Seq("ts", "ts_ntz").foreach { column =>
+ val selected = df.select(column)
+ assert(collect(selected.queryExecution.executedPlan) {
+ case _: CometNativeScanExec => true
+ }.nonEmpty)
+
+ val (sparkError, cometError) = checkSparkAnswerMaybeThrows(selected)
+ assert(Seq(sparkError, cometError).forall(_.exists(isOverflow)))
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+
+ test("filtered TIMESTAMP_MILLIS scans do not convert values Spark can skip") {
+ Seq(false, true).foreach { dictionaryEnabled =>
+ withTempDir { dir =>
+ val path = new Path(dir.toURI.toString, "part-r-0.parquet")
+ val schema = MessageTypeParser.parseMessageType("""
+ |message root {
+ | optional int32 id;
+ | optional int64 ts(TIMESTAMP_MILLIS);
+ | optional group s {
+ | optional int64 ts(TIMESTAMP_MILLIS);
+ | }
+ |}
+ |""".stripMargin)
+ val writer = createParquetWriter(schema, path, dictionaryEnabled)
+ (0 until 32).foreach { i =>
+ val record = new SimpleGroup(schema)
+ record.add(0, if (i % 2 == 0) 1 else 3)
+ // Milliseconds that overflow Long when converted to microseconds
+ record.add(1, 9223372036854776L)
+ record.addGroup(2).add(0, 9223372036854776L)
+ writer.write(record)
+ }
+ writer.close()
+
+ val footerReader = org.apache.parquet.hadoop.ParquetFileReader.open(
+ org.apache.parquet.hadoop.util.HadoopInputFile
+ .fromPath(path, spark.sessionState.newHadoopConf()))
+ try {
+ val idEncoding =
+ footerReader.getFooter.getBlocks.get(0).getColumns.get(0).getEncodingStats
+ assert(idEncoding.hasDictionaryEncodedPages == dictionaryEnabled)
+ assert(!dictionaryEnabled || !idEncoding.hasNonDictionaryEncodedPages)
+ } finally {
+ footerReader.close()
+ }
+
+ def timestampIn(size: Int): String =
+ (0 until size)
+ .map(i => f"timestamp'1970-01-01 00:00:$i%02d'")
+ .mkString("ts IN (", ", ", ")")
+
+ val predicates = Seq(
+ "ts < timestamp'1970-01-01 00:00:00'",
+ timestampIn(20),
+ timestampIn(21),
+ "ts <=> timestamp'1970-01-01 00:00:00'",
+ "s.ts < timestamp'1970-01-01 00:00:00'",
+ "id < 0") ++ (if (dictionaryEnabled) Seq("id = 2") else Seq.empty)
+ Seq(false, true).foreach { ansiEnabled =>
+ Seq(false, true).foreach { rowFilterPushdown =>
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString,
+ CometConf.COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED.key ->
+ rowFilterPushdown.toString) {
+ readParquetFile(path.toString) { df =>
+ predicates.foreach { predicate =>
+ val filtered = df.where(predicate)
+ assert(collect(filtered.queryExecution.executedPlan) {
+ case _: CometNativeScanExec => true
+ }.nonEmpty)
+ checkSparkAnswer(filtered)
+ }
+
+ if (CometConf.COMET_SCHEMA_EVOLUTION_ENABLED) {
+ val widenedSchema = StructType(
+ Seq(
+ StructField("id", LongType),
+ StructField("ts", TimestampType),
+ StructField("s", StructType(Seq(StructField("ts", TimestampType))))))
+ readParquetFile(path.toString, Some(widenedSchema)) { widened =>
+ checkSparkAnswer(widened.where("id < 0").select("ts"))
+ }
+ }
+ }
+ }
+ }
+ }
+
+ withSQLConf(
+ SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ "spark.comet.expression.IsNull.enabled" -> "false") {
+ readParquetFile(path.toString) { df =>
+ val filtered = df.where("id IS NULL").select("ts")
+ assert(collect(filtered.queryExecution.executedPlan) { case _: CometNativeScanExec =>
+ true
+ }.nonEmpty)
+ checkSparkAnswer(filtered)
+ }
+ }
+
+ if (isSpark40Plus && dictionaryEnabled) {
+ withTempView("timestamp_millis_lookup") {
+ Seq(2).toDF("k").createOrReplaceTempView("timestamp_millis_lookup")
+ withSQLConf("spark.comet.expression.EqualNullSafe.enabled" -> "false") {
+ readParquetFile(path.toString) { df =>
+ val filtered = df
+ .where("id <=> (SELECT max(k) FROM timestamp_millis_lookup)")
+ .select("ts")
+ assert(collect(filtered.queryExecution.executedPlan) {
+ case _: CometNativeScanExec => true
+ }.nonEmpty)
+ checkSparkAnswer(filtered)
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+
+ test("filtered TIMESTAMP_MILLIS scan preserves empty NOT IN null semantics") {
+ withTempDir { dir =>
+ val path = new Path(dir.toURI.toString, "part-r-0.parquet")
+ val schema = MessageTypeParser.parseMessageType("""
+ |message root {
+ | optional int64 ts(TIMESTAMP_MILLIS);
+ |}
+ |""".stripMargin)
+ val writer = createParquetWriter(schema, path)
+ writer.write(new SimpleGroup(schema))
+ val epoch = new SimpleGroup(schema)
+ epoch.add(0, 0L)
+ writer.write(epoch)
+ writer.close()
+
+ Seq(false, true).foreach { rowFilterPushdown =>
+ withSQLConf(
+ SQLConf.OPTIMIZER_EXCLUDED_RULES.key ->
+ Seq(ConvertToLocalRelation.ruleName, OptimizeIn.ruleName).mkString(","),
+ "spark.sql.legacy.nullInEmptyListBehavior" -> "false",
+ CometConf.COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED.key -> rowFilterPushdown.toString) {
+ readParquetFile(path.toString) { df =>
+ val filtered = df.filter(!df("ts").isin())
+ assert(collect(filtered.queryExecution.executedPlan) { case _: CometNativeScanExec =>
+ true
+ }.nonEmpty)
+ checkSparkAnswer(filtered)
+ }
+ }
+ }
+ }
+ }
+
test("timestamp as int96") {
import testImplicits._
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org