This is an automated email from the ASF dual-hosted git repository.
timsaucer pushed a commit to branch feat/ffi-with-extensions
in repository https://gitbox.apache.org/repos/asf/datafusion-python.git
commit 53453ea5bb7662f334bf8c2a098f3bafa6d383ca
Author: Tim Saucer <ti...@gmail.com>
AuthorDate: Fri Aug 7 14:19:06 2026 -0400
Document and test the context-outlives-DataFrame contract
A DataFrame does not keep its SessionContext alive. FFI components hold
a weak task-context provider, so operations that reach an FFI codec
after the context is collected fail with a clean out-of-scope error
rather than crashing. Lock that behavior in with a test and document
the ownership contract in the FFI guide and with_extensions docstring.
Co-Authored-By: Claude Fable 5 <no...@anthropic.com>
---
docs/source/contributor-guide/ffi.md | 7 +++++++
.../python/tests/_test_three_library_query_planner.py | 19 +++++++++++++++++++
python/datafusion/context.py | 5 +++++
3 files changed, 31 insertions(+)
diff --git a/docs/source/contributor-guide/ffi.md b/docs/source/contributor-guide/ffi.md
index 17c58826..a51e979e 100644
--- a/docs/source/contributor-guide/ffi.md
+++ b/docs/source/contributor-guide/ffi.md
@@ -327,6 +327,13 @@ cache bound components, and do not retain the context passed in. Catalogs are sh
with the source context, so registrations made during binding are not rolled back on
failure.
+The returned context is the strong owner of every installed component's task-context
+provider, and dependent objects do not extend its lifetime. A `DataFrame`, logical
+plan, or capsule can outlive the context, but any operation that reaches an FFI codec
+after the context is collected fails with `TaskContextProvider went out of scope over
+FFI boundary`. Keep the context alive for as long as objects derived from it are in
+use.
+
`MyPlannerExtension` in [`datafusion-ffi-query-planner-example`] is a complete Rust
implementation of this protocol, including extracting the task-context provider from
the supplied context and constructing a Python `SessionExtensionComponents`.
diff --git a/examples/datafusion-ffi-query-planner-example/python/tests/_test_three_library_query_planner.py b/examples/datafusion-ffi-query-planner-example/python/tests/_test_three_library_query_planner.py
index 5846661f..2fde3e2d 100644
--- a/examples/datafusion-ffi-query-planner-example/python/tests/_test_three_library_query_planner.py
+++ b/examples/datafusion-ffi-query-planner-example/python/tests/_test_three_library_query_planner.py
@@ -286,6 +286,25 @@ def test_with_extensions_failure_leaves_source_usable():
assert batches[0].column(0).to_pylist() == [0, 1, 2, 3, 4, 5]
+def test_dataframe_outliving_context_fails_cleanly():
+ """A DataFrame does not keep its SessionContext alive. FFI components
+ resolve the task context through a weak reference, so using the
+ DataFrame after dropping the context raises a clean error instead of
+ crashing. This locks in the documented ownership contract: the context
+ must outlive DataFrames that depend on FFI codecs."""
+ config = SessionConfig().with_extension(PlannerConfig(max_rows=2))
+ ctx = SessionContext(config).with_extensions(
+ ProviderCodecsExtension(), MyPlannerExtension()
+ )
+ ctx.register_table("numbers", MyTableProvider(1, 6, 1))
+ df = ctx.sql('SELECT "A" FROM numbers ORDER BY "A"')
+ del ctx
+ gc.collect()
+
+ with pytest.raises(Exception, match="went out of scope"):
+ df.collect()
+
+
def test_composed_codecs_with_query_planner():
"""A second pair of codecs installed on top of the provider codecs
composes with them instead of replacing them. The extra codecs
diff --git a/python/datafusion/context.py b/python/datafusion/context.py
index ce6bbd53..19b3191c 100644
--- a/python/datafusion/context.py
+++ b/python/datafusion/context.py
@@ -1880,6 +1880,11 @@ class SessionContext:
registering tables or otherwise mutating the context during binding is
not rolled back on failure.
+ The returned context is the strong owner of the installed components'
+ task-context providers. Keep it alive for as long as DataFrames or
+ plans derived from it are in use; FFI operations after the context is
+ collected raise an error.
+
Args:
extensions: One or more objects implementing
``__datafusion_session_extension__`` (see
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org