You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/08/10 21:33:51 UTC
(datafusion) branch gh-readonly-queue/main/pr-24094-a251b940882fd205a5bcfa3efd0d47367ac26bf7 created (now d5bd10d804)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-24094-a251b940882fd205a5bcfa3efd0d47367ac26bf7
in repository https://gitbox.apache.org/repos/asf/datafusion.git
at d5bd10d804 physical-plan: coerce UNION/INTERLEAVE schema mismatches at plan time (#24094)
This branch includes the following new commits:
new d5bd10d804 physical-plan: coerce UNION/INTERLEAVE schema mismatches at plan time (#24094)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion) 01/01: physical-plan: coerce UNION/INTERLEAVE schema mismatches at plan time (#24094)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-24094-a251b940882fd205a5bcfa3efd0d47367ac26bf7
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit d5bd10d8042772539630e7a00275c7b66cac95da
Author: dario curreri <48...@users.noreply.github.com>
AuthorDate: Mon Aug 10 23:33:27 2026 +0200
physical-plan: coerce UNION/INTERLEAVE schema mismatches at plan time (#24094)
## Which issue does this PR close?
Follow-up to #23861 (issue #15394). Not closing a new issue.
## Rationale for this change
#23861 fixed `UNION ALL` batches carrying the wrong nullability when one
leg
is `NOT NULL` and another isn't, by re-stamping each batch's schema
inside
`UnionExec`/`InterleaveExec`'s own `execute()`. In review, @alamb noted:
> ideally we could coerce the schema at plan time but I don't know how
to
> coerce nullability
This PR does that: the coercion becomes an explicit node in the plan
tree,
inserted when the plan is built, instead of invisible logic inside
`execute()`.
## What changes are included in this PR?
- Adds `CoerceSchemaExec`, a single-child passthrough `ExecutionPlan`
node.
`UnionExec::try_new`/`InterleaveExec::try_new` insert it above any child
whose own output schema disagrees with the computed union schema (in
practice, only nullability differs -- `UnionExec::try_new` already
rejects
real data-type mismatches via `calculate_union`).
- The actual batch re-stamping logic (`SchemaConformingStream`) is
unchanged;
it just lives under `CoerceSchemaExec::execute()` now instead of being
called directly from `UnionExec`/`InterleaveExec::execute()`.
- Because it's a real plan node, `CoerceSchemaExec` implements the full
`ExecutionPlan` surface a pure 1:1 passthrough needs to stay transparent
to the optimizer: statistics passthrough, filter/limit pushdown,
`benefits_from_input_partitioning() -> false` (so it doesn't trigger a
spurious repartition), and proto (de)serialization -- the node erases
itself on encode and is reconstructed by `try_new` on decode, so no
protobuf schema change was needed.
- A genuine data-type mismatch (as opposed to nullability-only) is now
rejected eagerly at plan-build time (via `EquivalenceProperties::
with_new_schema`) rather than lazily at `execute()`.
## Are these changes tested?
- New unit test
`test_union_partition_statistics_with_mismatched_nullability`
in `union.rs`, proving statistics aren't poisoned to `Absent` through
the
new node.
- Existing `union_nullable`/`union_nullable_spill` regression tests from
#23861 continue to pass unchanged.
- Updated the `sqllogictest` golden file (`union.slt`) where `EXPLAIN`
output now shows the new node for pre-existing nullability-mismatched
`UNION ALL` cases.
- Benchmarked against the previous (inline) approach: no measurable
performance difference in either the coerced or matched-schema case
(differences were within run-to-run noise).
## Are there any user-facing changes?
`EXPLAIN` output for a `UNION ALL`/interleaved plan with a nullability
mismatch across legs will now show a `CoerceSchemaExec` node that wasn't
there before. No behavioral or correctness change.
---------
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
---
datafusion/core/tests/dataframe/mod.rs | 3 +-
datafusion/physical-expr/src/projection.rs | 45 ++++
datafusion/physical-plan/src/union.rs | 259 ++++++++++++++++-------
datafusion/proto/tests/cases/plans/misc.rs | 107 +++++++++-
datafusion/sqllogictest/test_files/array_agg.slt | 6 +-
datafusion/sqllogictest/test_files/union.slt | 6 +-
6 files changed, 338 insertions(+), 88 deletions(-)
diff --git a/datafusion/core/tests/dataframe/mod.rs b/datafusion/core/tests/dataframe/mod.rs
index 73a9177ab7..a51c752c5c 100644
--- a/datafusion/core/tests/dataframe/mod.rs
+++ b/datafusion/core/tests/dataframe/mod.rs
@@ -7075,7 +7075,8 @@ async fn test_copy_to_preserves_order() -> Result<()> {
DataSinkExec: sink=CsvSink(file_groups=[])
SortExec: expr=[column1@0 DESC], preserve_partitioning=[false]
DataSourceExec: partitions=1, partition_sizes=[1]
- DataSourceExec: partitions=1, partition_sizes=[1]
+ ProjectionExec: expr=[CAST(column1@0 AS UInt64) as count]
+ DataSourceExec: partitions=1, partition_sizes=[1]
"
);
Ok(())
diff --git a/datafusion/physical-expr/src/projection.rs b/datafusion/physical-expr/src/projection.rs
index e3fd6ddf74..f8f4cf51fa 100644
--- a/datafusion/physical-expr/src/projection.rs
+++ b/datafusion/physical-expr/src/projection.rs
@@ -848,6 +848,22 @@ fn project_column_statistics_through_expr(
let inner_stats =
project_column_statistics_through_expr(cast_expr.expr.as_ref(), column_stats);
let target_type = cast_expr.cast_type();
+
+ // A cast whose source values are already of the target `DataType` never
+ // changes any value -- see `cast_array_by_name`'s same-type fast path in
+ // `ColumnarValue::cast_to`. In that case every statistic, not just
+ // min/max, carries over unchanged (this is what a cast that only
+ // re-stamps a column's nullability, as `UnionExec`/`InterleaveExec`
+ // insert, looks like here).
+ let already_target_type = matches!(
+ (inner_stats.min_value.get_value(), inner_stats.max_value.get_value()),
+ (Some(min), Some(max))
+ if min.data_type() == *target_type && max.data_type() == *target_type
+ );
+ if already_target_type {
+ return inner_stats;
+ }
+
ColumnStatistics {
min_value: inner_stats
.min_value
@@ -2934,6 +2950,35 @@ pub(crate) mod tests {
Ok(())
}
+ #[test]
+ fn test_project_statistics_with_same_type_cast_is_exact_passthrough() -> Result<()> {
+ // A cast to the column's own `DataType` (e.g. one that only re-stamps
+ // nullability via `CastExpr::new_with_target_field`, as `UnionExec`/
+ // `InterleaveExec` insert) never changes any value, so every
+ // statistic -- not just min/max -- should carry over unchanged.
+ let input_stats = get_stats();
+ let col0_stats = input_stats.column_statistics[0].clone();
+ let input_schema = get_schema();
+
+ let projection = ProjectionExprs::new(vec![ProjectionExpr {
+ expr: Arc::new(CastExpr::new(
+ Arc::new(Column::new("col0", 0)),
+ DataType::Int64,
+ None,
+ )),
+ alias: "casted".to_string(),
+ }]);
+
+ let output_stats = projection.project_statistics(
+ input_stats,
+ &projection.project_schema(&input_schema)?,
+ )?;
+
+ assert_eq!(output_stats.column_statistics[0], col0_stats);
+
+ Ok(())
+ }
+
#[test]
fn test_project_statistics_with_cast() -> Result<()> {
let input_stats = get_stats();
diff --git a/datafusion/physical-plan/src/union.rs b/datafusion/physical-plan/src/union.rs
index 88d2628ae2..fb62deecc3 100644
--- a/datafusion/physical-plan/src/union.rs
+++ b/datafusion/physical-plan/src/union.rs
@@ -42,104 +42,92 @@ use crate::filter_pushdown::{
FilterPushdownPropagation, PushedDown,
};
use crate::metrics::BaselineMetrics;
-use crate::projection::{ProjectionExec, make_with_child};
+use crate::projection::{ProjectionExec, ProjectionExpr, make_with_child};
use crate::statistics::{ChildStats, StatisticsArgs};
use crate::stream::ObservedStream;
-use arrow::array::RecordBatchOptions;
use arrow::datatypes::{Field, Schema, SchemaRef};
use arrow::record_batch::RecordBatch;
use datafusion_common::config::ConfigOptions;
use datafusion_common::stats::NdvFallback;
use datafusion_common::tree_node::TreeNodeRecursion;
use datafusion_common::{
- Result, assert_or_internal_err, exec_err, internal_datafusion_err,
+ Result, assert_or_internal_err, exec_err, internal_datafusion_err, plan_err,
};
use datafusion_execution::TaskContext;
+use datafusion_physical_expr::expressions::{CastExpr, Column};
use datafusion_physical_expr::{
EquivalenceProperties, PhysicalExpr, calculate_union, conjunction,
};
-use futures::{Stream, StreamExt};
+use futures::Stream;
use itertools::Itertools;
use log::{debug, trace, warn};
use tokio::macros::support::thread_rng_n;
-/// Wraps a child stream so that every batch it yields is re-stamped with
-/// `schema` instead of the child's own schema.
+/// Coerces `input`'s output schema to exactly `schema` via a `ProjectionExec`
+/// that re-stamps each column with the union's merged field (same
+/// `DataType`, but the union's merged nullability/name/metadata), or returns
+/// `input` unchanged if its schema already matches. [`UnionExec::try_new`]
+/// and [`InterleaveExec::try_new`] call this on every child, so the coercion
+/// is visible in the plan tree (e.g. in `EXPLAIN`) instead of happening
+/// invisibly inside the union operator's own `execute()`.
///
-/// This is used by both [`UnionExec`] and [`InterleaveExec`] when a child's
-/// output schema disagrees with the operator's declared output schema --
-/// in practice this only happens for nullability (the declared schema is
-/// nullable wherever *any* input's field is, but casts are only inserted
-/// between inputs when the *type* differs, not when only nullability
-/// does). For [`UnionExec`], [`UnionExec::try_new`] guarantees this: it
-/// calls `calculate_union`, which rejects any input whose field data types
-/// don't match the computed union schema. [`InterleaveExec::try_new`] does
-/// not repeat that check -- its inputs are only ever produced by the
-/// optimizer rewriting an already-validated `UnionExec`, whose children's
-/// types are therefore already known to agree -- but if this wrapper ever
-/// did see a genuine data type mismatch (e.g. from a hand-built
-/// `InterleaveExec`), `RecordBatch::try_new_with_options` below reports it
-/// as an error rather than silently yielding a corrupt batch.
-struct SchemaConformingStream {
- schema: SchemaRef,
- inner: SendableRecordBatchStream,
-}
-
-impl SchemaConformingStream {
- fn new(schema: SchemaRef, inner: SendableRecordBatchStream) -> Self {
- Self { schema, inner }
- }
-}
-
-impl RecordBatchStream for SchemaConformingStream {
- fn schema(&self) -> SchemaRef {
- Arc::clone(&self.schema)
- }
-}
-
-impl Stream for SchemaConformingStream {
- type Item = Result<RecordBatch>;
-
- fn poll_next(
- mut self: Pin<&mut Self>,
- cx: &mut Context<'_>,
- ) -> Poll<Option<Self::Item>> {
- self.inner.poll_next_unpin(cx).map(|opt| {
- opt.map(|batch_result| {
- batch_result.and_then(|batch| {
- let options =
- RecordBatchOptions::new().with_row_count(Some(batch.num_rows()));
- RecordBatch::try_new_with_options(
- Arc::clone(&self.schema),
- batch.columns().to_vec(),
- &options,
- )
- .map_err(Into::into)
- })
+/// A column whose `DataType` doesn't already match the union's is a genuine
+/// data type mismatch (as opposed to a nullability/name/metadata-only one),
+/// and is rejected eagerly here rather than silently cast or deferred to a
+/// runtime failure -- this only ever changes a column's declared schema,
+/// never its values.
+///
+/// Casting a column to its own `DataType` (only the `Field`'s nullability,
+/// name, or metadata changes) is a zero-copy relabeling: the cast kernel's
+/// same-type fast path (`cast_array_by_name`) just clones the `Arc<dyn
+/// Array>`, so this carries no runtime overhead over the schema it replaces.
+///
+/// See <https://github.com/apache/datafusion/issues/15394>.
+fn coerce_schema(
+ input: Arc<dyn ExecutionPlan>,
+ schema: &SchemaRef,
+) -> Result<Arc<dyn ExecutionPlan>> {
+ let input_schema = input.schema();
+ if &input_schema == schema {
+ return Ok(input);
+ }
+
+ let exprs = input_schema
+ .fields()
+ .iter()
+ .zip(schema.fields())
+ .enumerate()
+ .map(|(i, (input_field, target_field))| {
+ if input_field.data_type() != target_field.data_type() {
+ return plan_err!(
+ "UnionExec/InterleaveExec requires all inputs to have the same \
+ data type per column; column {i} has type {} in one input, but \
+ the union schema expects {}",
+ input_field.data_type(),
+ target_field.data_type()
+ );
+ }
+ let column: Arc<dyn PhysicalExpr> =
+ Arc::new(Column::new(input_field.name(), i));
+ let expr = if input_field == target_field {
+ column
+ } else {
+ Arc::new(CastExpr::new_with_target_field(
+ column,
+ Arc::clone(target_field),
+ None,
+ )) as Arc<dyn PhysicalExpr>
+ };
+ Ok(ProjectionExpr {
+ expr,
+ alias: target_field.name().clone(),
})
})
- }
+ .collect::<Result<Vec<_>>>()?;
- fn size_hint(&self) -> (usize, Option<usize>) {
- self.inner.size_hint()
- }
-}
-
-/// Wraps `stream` in a [`SchemaConformingStream`] if its schema disagrees
-/// with `schema`, otherwise returns it unchanged. See
-/// [`SchemaConformingStream`] and
-/// <https://github.com/apache/datafusion/issues/15394>.
-fn conform_stream_schema(
- schema: SchemaRef,
- stream: SendableRecordBatchStream,
-) -> SendableRecordBatchStream {
- if stream.schema() == schema {
- stream
- } else {
- Box::pin(SchemaConformingStream::new(schema, stream))
- }
+ Ok(Arc::new(ProjectionExec::try_new(exprs, input)?))
}
/// `UnionExec`: `UNION ALL` execution plan.
@@ -209,6 +197,10 @@ impl UnionExec {
// The schema of the inputs and the union schema is consistent when:
// - They have the same number of fields, and
// - Their fields have same types at the same indices.
+ let inputs = inputs
+ .into_iter()
+ .map(|input| coerce_schema(input, &schema))
+ .collect::<Result<Vec<_>>>()?;
let cache = Self::compute_properties(&inputs, schema)?;
Ok(Arc::new(UnionExec {
inputs,
@@ -380,7 +372,6 @@ impl ExecutionPlan for UnionExec {
if partition < input.output_partitioning().partition_count() {
let stream = input.execute(partition, context)?;
debug!("Found a Union partition to execute");
- let stream = conform_stream_schema(self.schema(), stream);
return Ok(Box::pin(ObservedStream::new(
stream,
baseline_metrics,
@@ -650,7 +641,12 @@ impl InterleaveExec {
can_interleave(inputs.iter()),
"Not all InterleaveExec children have a consistent hash or range partitioning"
);
- let cache = Self::compute_properties(&inputs)?;
+ let schema = union_schema(&inputs)?;
+ let inputs = inputs
+ .into_iter()
+ .map(|input| coerce_schema(input, &schema))
+ .collect::<Result<Vec<_>>>()?;
+ let cache = Self::compute_properties(&inputs, schema)?;
Ok(InterleaveExec {
inputs,
metrics: ExecutionPlanMetricsSet::new(),
@@ -664,8 +660,10 @@ impl InterleaveExec {
}
/// This function creates the cache object that stores the plan properties such as schema, equivalence properties, ordering, partitioning, etc.
- fn compute_properties(inputs: &[Arc<dyn ExecutionPlan>]) -> Result<PlanProperties> {
- let schema = union_schema(inputs)?;
+ fn compute_properties(
+ inputs: &[Arc<dyn ExecutionPlan>],
+ schema: SchemaRef,
+ ) -> Result<PlanProperties> {
let eq_properties = EquivalenceProperties::new(schema);
// Get output partitioning:
let output_partitioning = inputs[0].output_partitioning().clone();
@@ -763,7 +761,7 @@ impl ExecutionPlan for InterleaveExec {
for input in self.inputs.iter() {
if partition < input.output_partitioning().partition_count() {
let stream = input.execute(partition, Arc::clone(&context))?;
- input_stream_vec.push(conform_stream_schema(self.schema(), stream));
+ input_stream_vec.push(stream);
} else {
// Do not find a partition to execute
break;
@@ -1277,6 +1275,107 @@ mod tests {
Ok(())
}
+ #[test]
+ fn test_union_partition_statistics_with_mismatched_nullability() -> Result<()> {
+ // Regression test for the `ProjectionExec` wrapper `UnionExec::try_new`
+ // inserts above the non-nullable leg here (via `coerce_schema`):
+ // exact column statistics (min/max/null/distinct/sum/byte_size) must
+ // still make it through the wrapper's same-type `CastExpr`, not get
+ // poisoned into `Absent` the way a generic (type-changing) cast's
+ // statistics would be.
+ let (_, left, right, expected) = stats_merge_inputs();
+
+ // `total_byte_size` differs from the plain-merge fixture (52): the
+ // wrapper is a `ProjectionExec`, whose `statistics_from_inputs`
+ // recomputes `total_byte_size` from the (unchanged) schema's row
+ // width times row count, rather than trusting the wrapped leg's own
+ // self-reported total -- still `Exact`, just derived differently.
+ // left: 5 rows * 4 bytes (UInt32) = 20 (was 23); right is untouched
+ // (already nullable, so `coerce_schema` doesn't wrap it): 20 + 29 = 49.
+ let expected = expected.with_total_byte_size(Precision::Exact(49));
+
+ let non_nullable_schema =
+ Schema::new(vec![Field::new("a", DataType::UInt32, false)]);
+ let nullable_schema = Schema::new(vec![Field::new("a", DataType::UInt32, true)]);
+
+ let left: Arc<dyn ExecutionPlan> =
+ Arc::new(StatisticsExec::new(left, non_nullable_schema));
+ let right: Arc<dyn ExecutionPlan> =
+ Arc::new(StatisticsExec::new(right, nullable_schema));
+
+ let union = UnionExec::try_new(vec![left, right])?;
+ let stats =
+ StatisticsContext::new().compute(union.as_ref(), &StatisticsArgs::new())?;
+
+ assert_eq!(stats.as_ref(), &expected);
+ Ok(())
+ }
+
+ #[tokio::test]
+ async fn test_coerce_schema_no_op_when_already_matching() -> Result<()> {
+ let schema_not_null =
+ Arc::new(Schema::new(vec![Field::new("a", DataType::Int32, false)]));
+ let input: Arc<dyn ExecutionPlan> =
+ TestMemoryExec::try_new_exec(&[vec![]], Arc::clone(&schema_not_null), None)?;
+
+ let coerced = coerce_schema(Arc::clone(&input), &schema_not_null)?;
+ assert!(Arc::ptr_eq(&coerced, &input));
+
+ Ok(())
+ }
+
+ #[tokio::test]
+ async fn test_coerce_schema_casts_only_nullability() -> Result<()> {
+ // Mismatched nullability: the input gets wrapped in a `ProjectionExec`
+ // whose `CastExpr` re-stamps the column with the target's `Field`
+ // (same `DataType`, so this is a zero-copy relabeling, not a real cast).
+ let schema_not_null =
+ Arc::new(Schema::new(vec![Field::new("a", DataType::Int32, false)]));
+ let batch_not_null = RecordBatch::try_new(
+ Arc::clone(&schema_not_null),
+ vec![Arc::new(arrow::array::Int32Array::from(vec![1, 2]))],
+ )?;
+ let input: Arc<dyn ExecutionPlan> = TestMemoryExec::try_new_exec(
+ &[vec![batch_not_null]],
+ Arc::clone(&schema_not_null),
+ None,
+ )?;
+
+ let nullable_schema =
+ Arc::new(Schema::new(vec![Field::new("a", DataType::Int32, true)]));
+ let coerced = coerce_schema(Arc::clone(&input), &nullable_schema)?;
+ assert_eq!(&coerced.schema(), &nullable_schema);
+ let plan_str = crate::displayable(coerced.as_ref())
+ .indent(true)
+ .to_string();
+ assert!(
+ plan_str.contains("CAST"),
+ "expected a CAST in the coerced plan:\n{plan_str}"
+ );
+
+ let task_ctx = Arc::new(TaskContext::default());
+ let batches = collect(coerced, task_ctx).await?;
+ assert_eq!(batches.len(), 1);
+ assert_eq!(batches[0].schema(), nullable_schema);
+
+ Ok(())
+ }
+
+ #[test]
+ fn test_coerce_schema_rejects_genuine_type_mismatch() -> Result<()> {
+ let schema_int =
+ Arc::new(Schema::new(vec![Field::new("a", DataType::Int32, false)]));
+ let input: Arc<dyn ExecutionPlan> =
+ TestMemoryExec::try_new_exec(&[vec![]], Arc::clone(&schema_int), None)?;
+
+ let schema_utf8 =
+ Arc::new(Schema::new(vec![Field::new("a", DataType::Utf8, false)]));
+ let err = coerce_schema(input, &schema_utf8).unwrap_err();
+ assert!(err.to_string().contains("same data type per column"));
+
+ Ok(())
+ }
+
#[test]
fn test_interleave_partition_statistics_uses_shared_statistics_merge() -> Result<()> {
let (schema, left, right, expected) = stats_merge_inputs();
diff --git a/datafusion/proto/tests/cases/plans/misc.rs b/datafusion/proto/tests/cases/plans/misc.rs
index 254387f14e..41bb051c28 100644
--- a/datafusion/proto/tests/cases/plans/misc.rs
+++ b/datafusion/proto/tests/cases/plans/misc.rs
@@ -26,9 +26,10 @@ use datafusion::physical_expr::LexOrdering;
use datafusion::physical_plan::analyze::AnalyzeExec;
use datafusion::physical_plan::empty::EmptyExec;
use datafusion::physical_plan::explain::ExplainExec;
-use datafusion::physical_plan::expressions::{PhysicalSortExpr, col};
+use datafusion::physical_plan::expressions::{PhysicalSortExpr, col, lit};
use datafusion::physical_plan::metrics::MetricCategory;
use datafusion::physical_plan::placeholder_row::PlaceholderRowExec;
+use datafusion::physical_plan::projection::{ProjectionExec, ProjectionExpr};
use datafusion::physical_plan::repartition::RepartitionExec;
use datafusion::physical_plan::sorts::sort::SortExec;
use datafusion::physical_plan::union::{InterleaveExec, UnionExec};
@@ -174,6 +175,57 @@ fn roundtrip_union() -> Result<()> {
roundtrip_test(union)
}
+/// `UnionExec::try_new` coerces a nullability-mismatched leg by wrapping it
+/// in a `ProjectionExec` with a same-type `CastExpr` (see `coerce_schema` in
+/// `datafusion-physical-plan`'s `union` module) -- a zero-copy relabeling,
+/// not a real cast. `ProjectionExec` has an ordinary protobuf message, so
+/// unlike the node this replaced, there's no wrapper-erasure trick to verify;
+/// just that the decoded plan still contains the coercion and that its
+/// emitted batches expose the union's nullable schema.
+#[tokio::test]
+async fn roundtrip_union_with_mismatched_nullability_executes() -> Result<()> {
+ let literal_leg = |value: ScalarValue| -> Result<Arc<dyn ExecutionPlan>> {
+ Ok(Arc::new(ProjectionExec::try_new(
+ vec![ProjectionExpr {
+ expr: lit(value),
+ alias: "a".to_string(),
+ }],
+ Arc::new(PlaceholderRowExec::new(Arc::new(Schema::empty()))),
+ )?))
+ };
+ let non_nullable_leg = literal_leg(ScalarValue::Int64(Some(1)))?;
+ let nullable_leg = literal_leg(ScalarValue::Int64(None))?;
+
+ let union: Arc<dyn ExecutionPlan> =
+ UnionExec::try_new(vec![non_nullable_leg, nullable_leg])?;
+ assert!(union.schema().field(0).is_nullable());
+ assert!(
+ format!("{union:?}").contains("CastExpr"),
+ "expected a coercing CastExpr in plan:\n{union:?}"
+ );
+
+ let ctx = SessionContext::new();
+ let bytes = datafusion_proto::bytes::physical_plan_to_bytes(Arc::clone(&union))?;
+ let roundtripped = datafusion_proto::bytes::physical_plan_from_bytes(
+ bytes.as_ref(),
+ ctx.task_ctx().as_ref(),
+ )?;
+ assert!(roundtripped.schema().field(0).is_nullable());
+ assert!(
+ format!("{roundtripped:?}").contains("CastExpr"),
+ "expected a coercing CastExpr after roundtrip:\n{roundtripped:?}"
+ );
+
+ let batches =
+ datafusion::physical_plan::collect(roundtripped, ctx.task_ctx()).await?;
+ assert_eq!(batches.iter().map(|b| b.num_rows()).sum::<usize>(), 2);
+ for batch in &batches {
+ assert!(batch.schema().field(0).is_nullable());
+ }
+
+ Ok(())
+}
+
#[test]
fn roundtrip_repartition_preserve_order() -> Result<()> {
let field_a = Field::new("a", DataType::Int64, false);
@@ -320,6 +372,59 @@ fn roundtrip_interleave() -> Result<()> {
roundtrip_test(Arc::new(interleave))
}
+/// See [`roundtrip_union_with_mismatched_nullability_executes`]: the same
+/// wrapper-reinsertion behavior applies to `InterleaveExec::try_from_proto`.
+#[tokio::test]
+async fn roundtrip_interleave_with_mismatched_nullability_executes() -> Result<()> {
+ let partition = Partitioning::Hash(vec![], 3);
+ let literal_leg = |value: ScalarValue| -> Result<Arc<dyn ExecutionPlan>> {
+ let projection = ProjectionExec::try_new(
+ vec![ProjectionExpr {
+ expr: lit(value),
+ alias: "a".to_string(),
+ }],
+ Arc::new(PlaceholderRowExec::new(Arc::new(Schema::empty()))),
+ )?;
+ Ok(Arc::new(RepartitionExec::try_new(
+ Arc::new(projection),
+ partition.clone(),
+ )?))
+ };
+ let non_nullable_leg = literal_leg(ScalarValue::Int64(Some(1)))?;
+ let nullable_leg = literal_leg(ScalarValue::Int64(None))?;
+
+ let interleave: Arc<dyn ExecutionPlan> = Arc::new(InterleaveExec::try_new(vec![
+ non_nullable_leg,
+ nullable_leg,
+ ])?);
+ assert!(interleave.schema().field(0).is_nullable());
+ assert!(
+ format!("{interleave:?}").contains("CastExpr"),
+ "expected a coercing CastExpr in plan:\n{interleave:?}"
+ );
+
+ let ctx = SessionContext::new();
+ let bytes = datafusion_proto::bytes::physical_plan_to_bytes(Arc::clone(&interleave))?;
+ let roundtripped = datafusion_proto::bytes::physical_plan_from_bytes(
+ bytes.as_ref(),
+ ctx.task_ctx().as_ref(),
+ )?;
+ assert!(roundtripped.schema().field(0).is_nullable());
+ assert!(
+ format!("{roundtripped:?}").contains("CastExpr"),
+ "expected a coercing CastExpr after roundtrip:\n{roundtripped:?}"
+ );
+
+ let batches =
+ datafusion::physical_plan::collect(roundtripped, ctx.task_ctx()).await?;
+ assert_eq!(batches.iter().map(|b| b.num_rows()).sum::<usize>(), 2);
+ for batch in &batches {
+ assert!(batch.schema().field(0).is_nullable());
+ }
+
+ Ok(())
+}
+
#[test]
fn roundtrip_unnest() -> Result<()> {
let fa = Field::new("a", DataType::Int64, true);
diff --git a/datafusion/sqllogictest/test_files/array_agg.slt b/datafusion/sqllogictest/test_files/array_agg.slt
index f44e7f7d02..d5aaf8cab1 100644
--- a/datafusion/sqllogictest/test_files/array_agg.slt
+++ b/datafusion/sqllogictest/test_files/array_agg.slt
@@ -534,15 +534,15 @@ physical_plan
03)----RepartitionExec: partitioning=Hash([id@0], 4), input_partitions=5
04)------AggregateExec: mode=Partial, gby=[id@0 as id], aggr=[array_agg(DISTINCT a.foo), sum(DISTINCT Int64(1))], ordering_mode=Sorted
05)--------UnionExec
-06)----------ProjectionExec: expr=[1 as id, 2 as foo]
+06)----------ProjectionExec: expr=[1 as id, CAST(2 AS Int64) as foo]
07)------------PlaceholderRowExec
08)----------ProjectionExec: expr=[1 as id, NULL as foo]
09)------------PlaceholderRowExec
10)----------ProjectionExec: expr=[1 as id, NULL as foo]
11)------------PlaceholderRowExec
-12)----------ProjectionExec: expr=[1 as id, 3 as foo]
+12)----------ProjectionExec: expr=[1 as id, CAST(3 AS Int64) as foo]
13)------------PlaceholderRowExec
-14)----------ProjectionExec: expr=[1 as id, 2 as foo]
+14)----------ProjectionExec: expr=[1 as id, CAST(2 AS Int64) as foo]
15)------------PlaceholderRowExec
#######
diff --git a/datafusion/sqllogictest/test_files/union.slt b/datafusion/sqllogictest/test_files/union.slt
index cb5a06f729..d4776dd0c0 100644
--- a/datafusion/sqllogictest/test_files/union.slt
+++ b/datafusion/sqllogictest/test_files/union.slt
@@ -572,7 +572,7 @@ logical_plan
physical_plan
01)CoalescePartitionsExec: fetch=3
02)--UnionExec
-03)----ProjectionExec: expr=[count(Int64(1))@0 as cnt]
+03)----ProjectionExec: expr=[CAST(count(Int64(1))@0 AS Int64) as cnt]
04)------GlobalLimitExec: skip=0, fetch=3
05)--------AggregateExec: mode=Final, gby=[], aggr=[count(Int64(1))]
06)----------CoalescePartitionsExec
@@ -584,7 +584,7 @@ physical_plan
12)----------------------FilterExec: c13@1 != C2GT5KVyOPZpgKVl110TyZO0NcJ434, projection=[c1@0]
13)------------------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
14)--------------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/testing/data/csv/aggregate_test_100.csv]]}, projection=[c1, c13], file_type=csv, has_header=true
-15)----ProjectionExec: expr=[1 as cnt]
+15)----ProjectionExec: expr=[CAST(1 AS Int64) as cnt]
16)------PlaceholderRowExec
17)----ProjectionExec: expr=[lead(b.c1,Int64(1)) ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING@1 as cnt]
18)------GlobalLimitExec: skip=0, fetch=3
@@ -721,7 +721,7 @@ logical_plan
11)----------EmptyRelation: rows=1
physical_plan
01)UnionExec
-02)--ProjectionExec: expr=[count(Int64(1))@1 as count, n@0 as n]
+02)--ProjectionExec: expr=[count(Int64(1))@1 as count, CAST(n@0 AS Int64) as n]
03)----AggregateExec: mode=SinglePartitioned, gby=[n@0 as n], aggr=[count(Int64(1))], ordering_mode=Sorted
04)------ProjectionExec: expr=[5 as n]
05)--------PlaceholderRowExec
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org