You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/06/24 22:09:36 UTC
(datafusion) branch gh-readonly-queue/main/pr-23029-1ed57cb82dadf304a69d30fbddd4da6ed706c22b created (now 2b64c9fb49)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-23029-1ed57cb82dadf304a69d30fbddd4da6ed706c22b
in repository https://gitbox.apache.org/repos/asf/datafusion.git
at 2b64c9fb49 [physical-plan]: remove deprecated spill_record_batch_by_size (#23029)
This branch includes the following new commits:
new 2b64c9fb49 [physical-plan]: remove deprecated spill_record_batch_by_size (#23029)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion) 01/01: [physical-plan]: remove deprecated spill_record_batch_by_size (#23029)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-23029-1ed57cb82dadf304a69d30fbddd4da6ed706c22b
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit 2b64c9fb496058f35568143c3475ce2079d4fc8b
Author: Andrew Lamb <an...@nerdnetworks.org>
AuthorDate: Wed Jun 24 18:09:14 2026 -0400
[physical-plan]: remove deprecated spill_record_batch_by_size (#23029)
## Which issue does this PR close?
- part of https://github.com/apache/datafusion/issues/23080
## Rationale for this change
`datafusion_physical_plan::spill::spill_record_batch_by_size` was
deprecated in DataFusion `46.0.0` in favor of
`SpillManager::spill_record_batch_by_size`.
The [API health policy deprecation
guidelines](https://datafusion.apache.org/contributor-guide/api-health.html#deprecation-guidelines)
say deprecated methods remain for 6 major versions or 6 months,
whichever is longer. This API has exceeded that window, so this removes
the deprecated wrapper.
## What changes are included in this PR?
- Removes the deprecated
`datafusion_physical_plan::spill::spill_record_batch_by_size` function.
## Are these changes tested?
By CI
## Are there any user-facing changes?
Yes. This removes a public Rust API that was deprecated in DataFusion
`46.0.0`. Downstream users should migrate to
`datafusion_physical_plan::spill::SpillManager::spill_record_batch_by_size`.
This is an API change and should be labeled `api-change`.
---
datafusion/physical-plan/src/spill/mod.rs | 31 +---------------------
docs/source/library-user-guide/upgrading/55.0.0.md | 8 ++++++
2 files changed, 9 insertions(+), 30 deletions(-)
diff --git a/datafusion/physical-plan/src/spill/mod.rs b/datafusion/physical-plan/src/spill/mod.rs
index 3c95a1da5b..00c9ac0631 100644
--- a/datafusion/physical-plan/src/spill/mod.rs
+++ b/datafusion/physical-plan/src/spill/mod.rs
@@ -30,7 +30,7 @@ pub use spill_manager::SpillManager;
use std::fs::File;
use std::io::BufReader;
-use std::path::{Path, PathBuf};
+use std::path::Path;
use std::pin::Pin;
use std::sync::Arc;
use std::task::{Context, Poll};
@@ -245,35 +245,6 @@ impl RecordBatchStream for SpillReaderStream {
}
}
-/// Spill the `RecordBatch` to disk as smaller batches
-/// split by `batch_size_rows`
-#[deprecated(
- since = "46.0.0",
- note = "This method is deprecated. Use `SpillManager::spill_record_batch_by_size` instead."
-)]
-#[expect(clippy::needless_pass_by_value)]
-pub fn spill_record_batch_by_size(
- batch: &RecordBatch,
- path: PathBuf,
- schema: SchemaRef,
- batch_size_rows: usize,
-) -> Result<()> {
- let mut offset = 0;
- let total_rows = batch.num_rows();
- let mut writer =
- IPCStreamWriter::new(&path, schema.as_ref(), SpillCompression::Uncompressed)?;
-
- while offset < total_rows {
- let length = std::cmp::min(total_rows - offset, batch_size_rows);
- let batch = batch.slice(offset, length);
- offset += batch.num_rows();
- writer.write(&batch)?;
- }
- writer.finish()?;
-
- Ok(())
-}
-
/// Write in Arrow IPC Stream format to a file.
///
/// Stream format is used for spill because it supports dictionary replacement, and the random
diff --git a/docs/source/library-user-guide/upgrading/55.0.0.md b/docs/source/library-user-guide/upgrading/55.0.0.md
index 6d1f834abf..d0778a3619 100644
--- a/docs/source/library-user-guide/upgrading/55.0.0.md
+++ b/docs/source/library-user-guide/upgrading/55.0.0.md
@@ -30,6 +30,14 @@ to the main branch and are awaiting release in this version.
`datafusion_common::config::Dialect::AVAILABLE` has been removed. Use
`Dialect::available()` instead.
+### `spill_record_batch_by_size` removed
+
+`datafusion_physical_plan::spill::spill_record_batch_by_size` has been removed.
+This function was deprecated in DataFusion `46.0.0`.
+
+Use `datafusion_physical_plan::spill::SpillManager::spill_record_batch_by_size`
+instead.
+
### Decimal scalar formatting uses human-readable values
Decimal scalar literals in `EXPLAIN` output, expression display strings, and
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org