You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/14 17:09:55 UTC
(datafusion-ballista) branch asf-site updated: Publish built docs triggered by e7ce834869e2825d78e3e9790f0993fbd33b8f90
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
The following commit(s) were added to refs/heads/asf-site by this push:
new c51e25449 Publish built docs triggered by e7ce834869e2825d78e3e9790f0993fbd33b8f90
c51e25449 is described below
commit c51e254497b720b6af0b26b1a997d56d011e65eb
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Wed Oct 14 17:09:50 2026 +0000
Publish built docs triggered by e7ce834869e2825d78e3e9790f0993fbd33b8f90
---
_sources/contributors-guide/dataframe-cache.md.txt | 221 +++++++++++
_sources/index.rst.txt | 1 +
changelog/0.10.0.html | 1 +
changelog/0.11.0.html | 1 +
changelog/0.12.0.html | 1 +
changelog/0.5.0.html | 1 +
changelog/0.6.0.html | 1 +
changelog/0.7.0.html | 1 +
changelog/0.8.0.html | 1 +
changelog/0.9.0.html | 1 +
changelog/43.0.0.html | 1 +
changelog/44.0.0.html | 1 +
changelog/45.0.0.html | 1 +
changelog/46.0.0.html | 1 +
changelog/47.0.0.html | 1 +
changelog/48.0.0.html | 1 +
changelog/49.0.0.html | 1 +
changelog/50.0.0.html | 1 +
changelog/51.0.0.html | 1 +
changelog/52.0.0.html | 1 +
changelog/53.0.0.html | 1 +
changelog/54.0.0.html | 1 +
changelog/54.1.0.html | 1 +
changelog/55.0.0.html | 1 +
changelog/index.html | 1 +
community/adopters.html | 1 +
community/communication.html | 1 +
contributors-guide/architecture.html | 1 +
contributors-guide/benchmarking/ballista.html | 1 +
contributors-guide/benchmarking/comet.html | 1 +
contributors-guide/benchmarking/environment.html | 1 +
contributors-guide/benchmarking/index.html | 1 +
contributors-guide/benchmarking/spark.html | 1 +
contributors-guide/benchmarking/trino.html | 1 +
contributors-guide/code-organization.html | 7 +-
.../{development.html => dataframe-cache.html} | 411 ++++++++++-----------
contributors-guide/development.html | 1 +
contributors-guide/shuffle.html | 7 +-
contributors-guide/user-personas.html | 1 +
genindex.html | 1 +
index.html | 2 +
objects.inv | Bin 1327 -> 1367 bytes
search.html | 1 +
searchindex.js | 2 +-
upgrading/54.0.0.html | 1 +
upgrading/55.0.0.html | 1 +
upgrading/index.html | 1 +
user-guide/cli.html | 1 +
user-guide/configs.html | 1 +
user-guide/deployment/cargo-install.html | 1 +
user-guide/deployment/docker-compose.html | 1 +
user-guide/deployment/docker.html | 1 +
user-guide/deployment/index.html | 1 +
user-guide/deployment/kubernetes.html | 1 +
user-guide/deployment/quick-start.html | 1 +
user-guide/extending-components.html | 1 +
user-guide/extensions-example.html | 1 +
user-guide/faq.html | 1 +
user-guide/flightsql.html | 1 +
user-guide/history-server.html | 1 +
user-guide/introduction.html | 1 +
user-guide/metrics.html | 1 +
user-guide/python/cloud-storage.html | 1 +
user-guide/python/index.html | 1 +
user-guide/python/jupyter.html | 1 +
user-guide/python/quickstart.html | 1 +
user-guide/rust.html | 1 +
user-guide/scheduler.html | 1 +
user-guide/tuning-guide.html | 1 +
69 files changed, 497 insertions(+), 215 deletions(-)
diff --git a/_sources/contributors-guide/dataframe-cache.md.txt b/_sources/contributors-guide/dataframe-cache.md.txt
new file mode 100644
index 000000000..4381bded9
--- /dev/null
+++ b/_sources/contributors-guide/dataframe-cache.md.txt
@@ -0,0 +1,221 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# DataFrame Cache Design (`DataFrame::cache()`)
+
+## Goal
+
+`df.cache()` marks a DataFrame to be **computed once and reused** by later
+actions instead of recomputed. In Ballista a cached dataset is **shuffle output
+that is kept (pinned) past normal job cleanup, plus a scheduler-side record of
+where that output lives**. It reuses the native [blocking shuffle](shuffle.md)
+as its storage and transport — no separate cache store, no new wire format, and
+the read-back is an ordinary `ShuffleReaderExec`.
+
+The cache is a **hint, not a correctness dependency**: a cached node keeps its
+input subplan (lineage), so lost cache data is simply recomputed — never a wrong
+or missing answer.
+
+## The model
+
+In the **logical plan**, a cached dataset is a `BallistaCacheNode` wrapping the
+subplan to cache. Materialized, it has two pieces:
+
+1. **Pinned shuffle output** — the cached subplan runs as its own job; its final
+ stage's shuffle files (`{work_dir}/{job_id}/{stage_id}/…`) survive job cleanup.
+2. **A cross-job registry** — scheduler state (the `CacheState` trait, in-memory
+ `InMemoryCacheState` over `CacheRegistry`) mapping a cache key to the schema
+ and `PartitionLocation`s of that output, so a _different_ job can read it.
+
+A registry is needed because Ballista materializes the cache in a separate
+distributed **job**: the mapping from a cache to its shuffle output must outlive
+that job and be visible to other jobs, so the scheduler owns it explicitly.
+
+**Keying:** `(session_id, cache_id)`, where `cache_id` is a per-`cache()`-call
+UUID in the `BallistaCacheNode`. Reuse happens whenever a submitted plan still
+contains that node (same `cache_id`) — the cached DataFrame reused directly or
+built upon. Dedup across _different_ plans with equal content is deferred.
+
+## End-to-end flow
+
+Three roles: the **client** holds the plan, the **scheduler** owns planning and
+the registry, and the **executors** run tasks and hold the shuffle files.
+
+`df.cache()` is lazy: it returns a DataFrame whose **logical plan** is wrapped in
+a `BallistaCacheNode` with a fresh `cache_id` — nothing executes yet. Reuse is
+keyed on that `cache_id`: any plan later submitted that still contains the node —
+the same cached DataFrame, or a query built on top of it — resolves to the same
+**cache entry** (the registry's record of the materialized output). Calling
+`df.cache()` again mints a new `cache_id`.
+
+Miss and hit are not separate stories — they **converge**: a miss materializes
+and then reads the pinned output, while a hit does the same read and skips the
+materialize.
+
+```text
+ df.cache() client; lazy — wraps the logical
+ │ plan in BallistaCacheNode
+ ▼ {cache_id, session_id, input}
+ submit a plan carrying the node client ──► scheduler
+ │
+ ▼
+ lookup((session_id, cache_id)) scheduler
+ │
+ ┌────────────┴─────────────┐
+ MISS HIT
+ │ │
+ ▼ ▼
+ begin_materialization validate locations
+ run `input` as its own against live executors
+ job (two-job split, #1993) (any dead ⇒ re-MISS)
+ → executors write the │
+ pinned shuffle files │
+ │ │
+ ▼ │
+ complete_materialization │
+ → Materialized (fresh locs) │
+ │ │
+ └────────────┬──────────────┘
+ ▼
+ lower BallistaCacheNode ──► physical ShuffleReaderExec,
+ reading the pinned output served by the executors
+ │
+ ▼
+ results ──► client
+```
+
+**Cold path — first action (miss → materialize):**
+
+1. The client submits a plan containing `BallistaCacheNode{cache_id, session_id, input}`.
+2. The scheduler forms `key = (session_id, cache_id)` and calls `lookup(key)` → miss.
+3. `begin_materialization(key, job)` claims the key; the scheduler runs `input`
+ as its **own job** (a logical-plan rewrite that splits the query into two
+ jobs, see #1993), writing pinned shuffle output on the executors.
+4. On that job's `JobFinished`, `complete_materialization(key, job, schema, locations)` records the output → the cache entry becomes `Materialized`.
+5. The outer query reads that output and returns results — so the first action
+ both materializes and reads back.
+
+**Warm path — later action (hit → read-back):**
+
+1. The client submits a plan that still carries the same `cache_id`.
+2. `lookup(key)` → hit. Its locations are validated against the live-executor
+ set; a dead location downgrades the hit to a miss (see [Invalidation](#invalidation)).
+3. The scheduler lowers the `BallistaCacheNode` to a **physical**
+ `ShuffleReaderExec` over the recorded locations — the same mechanism stages
+ use to read each other — so `input` is never planned or executed.
+4. Results return with no materialization job.
+
+**Ordering & concurrency.** Within one action the read is sequenced after the
+create: the read's `ShuffleReaderExec` needs the recorded locations, so the
+scheduler resolves it only on the create job's `JobFinished` — event-driven, no
+polling. A _concurrent_ action that finds the key
+`Pending` (`begin_materialization` → `AlreadyPending`) must either wait for that
+materialization or fall back to computing the subplan inline; the exact policy —
+wait-and-notify vs. poll vs. inline, and releasing waiters if the job aborts — is
+an open question.
+
+## Registry lifecycle
+
+```text
+ (absent) --begin--> Pending{job} --complete(job)--> Materialized
+ Pending{job} --abort(job)-----> (absent)
+ Materialized --invalidate------> (absent)
+```
+
+The contract is safe against the out-of-order events a distributed scheduler
+produces (a completion callback racing invalidation, teardown, or a re-claim):
+
+- `begin_materialization` atomically gives one job the claim.
+- `complete_materialization` promotes **only** a `Pending` entry owned by that
+ job (duplicate completions are idempotent); anything else is `Stale`, so a
+ late completion cannot resurrect a removed entry or overwrite a newer claim.
+- `abort_materialization` clears a failed job's pending claim.
+- `invalidate` / `invalidate_executor` / `remove_session` remove entries and
+ **return them** so the caller can unpin the shuffle files (Ballista deletes
+ explicitly — there is no reclaimer).
+- `pinned_stages()` lists the `(job_id, stage_id)` stages to protect from cleanup.
+
+## Pinning
+
+Shuffle output is normally reclaimed after a job finishes. A cached stage must
+be exempted, so job-data cleanup consults `pinned_stages()` and skips those
+stage dirs. Pins are per-`(job_id, stage_id)` because shuffle data is stored and
+deleted per stage, and a cached stage is often an _intermediate_ stage (reaped
+immediately on job success) that a job-level pin could not protect.
+
+## Invalidation
+
+An entry is removed when its data is no longer trustworthy or needed:
+
+- **Executor loss** (`invalidate_executor`) — any materialized entry with a
+ partition on the lost executor is dropped.
+- **Session teardown** (`remove_session`) — all of a session's entries are
+ dropped.
+- **Explicit / eviction** (`invalidate`) — a single entry is dropped (used by
+ the future eviction sweeper).
+
+Invalidation is **whole-entry**: the entire cache is dropped and re-materialized
+in full on the next access — we never serve a cache with missing partitions
+(per-partition rebuild is a future optimization). Because a cached node keeps its
+lineage, invalidation always degrades to a recompute, never a wrong answer.
+
+Each removal **returns the removed entries** so the caller can unpin and delete
+their shuffle files — Ballista has no background reclaimer, so cleanup is
+explicit. A removed entry's next `lookup` is a miss, which re-triggers
+materialization.
+
+> **Executor loss during materialization (executor-loss follow-up).** If an
+> executor is lost after the cached stage finishes but before its job completes,
+> `complete_materialization` can record a now-dead location (the in-graph
+> recovery only rebuilds output an unfinished stage still reads). The fix is
+> **validate-on-read**: on a hit the planner checks locations against the
+> live-executor set and, if any is dead, treats it as a miss and `invalidate`s
+> it — enforcing "never read from a dead executor" under all timings.
+
+## Durability & reconciliation
+
+`CacheState` is a pluggable backend; the in-memory one loses entries on scheduler
+restart (caches become orphaned files, reclaimed by cleanup), and a durable
+backend can persist and rehydrate them — an optional efficiency extension. A
+stale or missing location is caught by validate-on-read (executor gone) or the
+`ShuffleReader` fetch-failure path (file gone) and degrades to re-materialization.
+
+## Future work
+
+- **Eviction:** entries today leave only via invalidation/executor-loss/session
+ teardown, so a long-running scheduler accumulates cache data. Planned: a TTL /
+ size-cap sweeper (via `invalidate`) and/or idle-session expiry (via
+ `remove_session`), using the removal-returns-entries contract above.
+- **Locality:** place read-back tasks on the executor holding each partition
+ (`TaskDistributionPolicy::Custom`), Flight fetch as fallback (see [#2319]).
+
+## Implementation plan
+
+1. **Registry** (done) — `CacheRegistry` + `CacheState`/`InMemoryCacheState`.
+2. **Materialize-on-miss + read-back** — wire `lookup`/`begin`/`complete`/`abort`
+ into planning and the job handlers; builds on the two-job rewrite in [#1993].
+3. **Pinning** — teach job-data cleanup to honor `pinned_stages()`.
+4. **Locality** — place read-back tasks where partitions live ([#2319]).
+5. **Executor-loss invalidation + session teardown** — wire
+ `invalidate_executor` / `remove_session` and the unpin path, add
+ validate-on-read on the hit path, and enable an end-to-end no-recompute test.
+6. **Eviction** — TTL / size cap / idle-session expiry.
+
+[#1993]: https://github.com/apache/datafusion-ballista/pull/1993
+[#2319]: https://github.com/apache/datafusion-ballista/issues/2319
diff --git a/_sources/index.rst.txt b/_sources/index.rst.txt
index aeeb46a1b..06ee8223b 100644
--- a/_sources/index.rst.txt
+++ b/_sources/index.rst.txt
@@ -79,6 +79,7 @@ Table of content
contributors-guide/architecture
contributors-guide/shuffle
+ contributors-guide/dataframe-cache
contributors-guide/code-organization
contributors-guide/user-personas
contributors-guide/development
diff --git a/changelog/0.10.0.html b/changelog/0.10.0.html
index b9b04798f..324e1ceb9 100644
--- a/changelog/0.10.0.html
+++ b/changelog/0.10.0.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/changelog/0.11.0.html b/changelog/0.11.0.html
index 5c8bca3b4..a484f0606 100644
--- a/changelog/0.11.0.html
+++ b/changelog/0.11.0.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/changelog/0.12.0.html b/changelog/0.12.0.html
index 642af1ed1..dda7d8056 100644
--- a/changelog/0.12.0.html
+++ b/changelog/0.12.0.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/changelog/0.5.0.html b/changelog/0.5.0.html
index 02d624ac4..4bacd3850 100644
--- a/changelog/0.5.0.html
+++ b/changelog/0.5.0.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/changelog/0.6.0.html b/changelog/0.6.0.html
index a557f7703..e852c3144 100644
--- a/changelog/0.6.0.html
+++ b/changelog/0.6.0.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/changelog/0.7.0.html b/changelog/0.7.0.html
index 90f47b40e..53790f168 100644
--- a/changelog/0.7.0.html
+++ b/changelog/0.7.0.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/changelog/0.8.0.html b/changelog/0.8.0.html
index 9a7d5c614..2b63ca42f 100644
--- a/changelog/0.8.0.html
+++ b/changelog/0.8.0.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/changelog/0.9.0.html b/changelog/0.9.0.html
index 2855838aa..cd216be13 100644
--- a/changelog/0.9.0.html
+++ b/changelog/0.9.0.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/changelog/43.0.0.html b/changelog/43.0.0.html
index 601d191e6..e297628fd 100644
--- a/changelog/43.0.0.html
+++ b/changelog/43.0.0.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/changelog/44.0.0.html b/changelog/44.0.0.html
index 82bcafefa..16a3b511d 100644
--- a/changelog/44.0.0.html
+++ b/changelog/44.0.0.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/changelog/45.0.0.html b/changelog/45.0.0.html
index ddac1cd53..1d3d81101 100644
--- a/changelog/45.0.0.html
+++ b/changelog/45.0.0.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/changelog/46.0.0.html b/changelog/46.0.0.html
index 290c668d1..cd873a5aa 100644
--- a/changelog/46.0.0.html
+++ b/changelog/46.0.0.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/changelog/47.0.0.html b/changelog/47.0.0.html
index 3bc2bbdb4..0585b81d2 100644
--- a/changelog/47.0.0.html
+++ b/changelog/47.0.0.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/changelog/48.0.0.html b/changelog/48.0.0.html
index 35d7a51a6..23903a03c 100644
--- a/changelog/48.0.0.html
+++ b/changelog/48.0.0.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/changelog/49.0.0.html b/changelog/49.0.0.html
index c092087c0..308ddd384 100644
--- a/changelog/49.0.0.html
+++ b/changelog/49.0.0.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/changelog/50.0.0.html b/changelog/50.0.0.html
index dd6cd191c..e9bee1b63 100644
--- a/changelog/50.0.0.html
+++ b/changelog/50.0.0.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/changelog/51.0.0.html b/changelog/51.0.0.html
index 859fb202d..36d0e8825 100644
--- a/changelog/51.0.0.html
+++ b/changelog/51.0.0.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/changelog/52.0.0.html b/changelog/52.0.0.html
index 2032dd659..0ca54877a 100644
--- a/changelog/52.0.0.html
+++ b/changelog/52.0.0.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/changelog/53.0.0.html b/changelog/53.0.0.html
index 1826259d8..0ca9d7ce5 100644
--- a/changelog/53.0.0.html
+++ b/changelog/53.0.0.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/changelog/54.0.0.html b/changelog/54.0.0.html
index 1d31aa5f2..fbb66574d 100644
--- a/changelog/54.0.0.html
+++ b/changelog/54.0.0.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/changelog/54.1.0.html b/changelog/54.1.0.html
index 21330f166..040a18140 100644
--- a/changelog/54.1.0.html
+++ b/changelog/54.1.0.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/changelog/55.0.0.html b/changelog/55.0.0.html
index 037a18df7..85d9d8087 100644
--- a/changelog/55.0.0.html
+++ b/changelog/55.0.0.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/changelog/index.html b/changelog/index.html
index ca7c70ad9..38e898d29 100644
--- a/changelog/index.html
+++ b/changelog/index.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/community/adopters.html b/community/adopters.html
index 315eff59d..181dd3656 100644
--- a/community/adopters.html
+++ b/community/adopters.html
@@ -304,6 +304,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/community/communication.html b/community/communication.html
index 9dd3e5f77..6f4023e93 100644
--- a/community/communication.html
+++ b/community/communication.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/contributors-guide/architecture.html b/contributors-guide/architecture.html
index 8af023262..64c1ca071 100644
--- a/contributors-guide/architecture.html
+++ b/contributors-guide/architecture.html
@@ -305,6 +305,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active"><a aria-current="page" class="current reference internal" href="#">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="development.html">Ballista Development</a></li>
diff --git a/contributors-guide/benchmarking/ballista.html b/contributors-guide/benchmarking/ballista.html
index ec9d57aba..b85f785a6 100644
--- a/contributors-guide/benchmarking/ballista.html
+++ b/contributors-guide/benchmarking/ballista.html
@@ -305,6 +305,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../development.html">Ballista Development</a></li>
diff --git a/contributors-guide/benchmarking/comet.html b/contributors-guide/benchmarking/comet.html
index b8d3e9242..373724351 100644
--- a/contributors-guide/benchmarking/comet.html
+++ b/contributors-guide/benchmarking/comet.html
@@ -305,6 +305,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../development.html">Ballista Development</a></li>
diff --git a/contributors-guide/benchmarking/environment.html b/contributors-guide/benchmarking/environment.html
index d6c17bfc5..b4997d85d 100644
--- a/contributors-guide/benchmarking/environment.html
+++ b/contributors-guide/benchmarking/environment.html
@@ -305,6 +305,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../development.html">Ballista Development</a></li>
diff --git a/contributors-guide/benchmarking/index.html b/contributors-guide/benchmarking/index.html
index 6b7401c75..9c5c365eb 100644
--- a/contributors-guide/benchmarking/index.html
+++ b/contributors-guide/benchmarking/index.html
@@ -305,6 +305,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../development.html">Ballista Development</a></li>
diff --git a/contributors-guide/benchmarking/spark.html b/contributors-guide/benchmarking/spark.html
index 83306494a..4b1851524 100644
--- a/contributors-guide/benchmarking/spark.html
+++ b/contributors-guide/benchmarking/spark.html
@@ -305,6 +305,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../development.html">Ballista Development</a></li>
diff --git a/contributors-guide/benchmarking/trino.html b/contributors-guide/benchmarking/trino.html
index 3b92d6d92..1b4ff1f21 100644
--- a/contributors-guide/benchmarking/trino.html
+++ b/contributors-guide/benchmarking/trino.html
@@ -305,6 +305,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../development.html">Ballista Development</a></li>
diff --git a/contributors-guide/code-organization.html b/contributors-guide/code-organization.html
index dec89b801..46f0ac618 100644
--- a/contributors-guide/code-organization.html
+++ b/contributors-guide/code-organization.html
@@ -46,7 +46,7 @@
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="User Personas" href="user-personas.html" />
- <link rel="prev" title="Shuffle Design" href="shuffle.html" />
+ <link rel="prev" title="DataFrame Cache Design (DataFrame::cache())" href="dataframe-cache.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="color-scheme" content="light dark"/>
<meta name="docsearch:language" content="en"/>
@@ -305,6 +305,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1 current active"><a aria-current="page" class="current reference internal" href="#">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="development.html">Ballista Development</a></li>
@@ -457,12 +458,12 @@
<div class="prev-next-area">
<a class="left-prev"
- href="shuffle.html"
+ href="dataframe-cache.html"
title="previous page">
<i class="fa-solid fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Shuffle Design</p>
+ <p class="prev-next-title">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</p>
</div>
</a>
<a class="right-next"
diff --git a/contributors-guide/development.html b/contributors-guide/dataframe-cache.html
similarity index 54%
copy from contributors-guide/development.html
copy to contributors-guide/dataframe-cache.html
index 32e03c3b1..9f945f3a4 100644
--- a/contributors-guide/development.html
+++ b/contributors-guide/dataframe-cache.html
@@ -7,7 +7,7 @@
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Ballista Development — Apache DataFusion Ballista documentation</title>
+ <title>DataFrame Cache Design (DataFrame::cache()) — Apache DataFusion Ballista documentation</title>
@@ -41,12 +41,12 @@
<script src="../_static/documentation_options.js?v=5929fcd5"></script>
<script src="../_static/doctools.js?v=fd6eb6e6"></script>
<script src="../_static/sphinx_highlight.js?v=6ffebe34"></script>
- <script>DOCUMENTATION_OPTIONS.pagename = 'contributors-guide/development';</script>
+ <script>DOCUMENTATION_OPTIONS.pagename = 'contributors-guide/dataframe-cache';</script>
<script>DOCUMENTATION_OPTIONS.search_as_you_type = false;</script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
- <link rel="next" title="Benchmarking" href="benchmarking/index.html" />
- <link rel="prev" title="User Personas" href="user-personas.html" />
+ <link rel="next" title="Ballista Code Organization" href="code-organization.html" />
+ <link rel="prev" title="Shuffle Design" href="shuffle.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="color-scheme" content="light dark"/>
<meta name="docsearch:language" content="en"/>
@@ -305,9 +305,10 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1 current active"><a aria-current="page" class="current reference internal" href="#">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="user-personas.html">User Personas</a></li>
-<li class="toctree-l1 current active"><a aria-current="page" class="current reference internal" href="#">Ballista Development</a></li>
+<li class="toctree-l1"><a class="reference internal" href="development.html">Ballista Development</a></li>
<li class="toctree-l1 has-children"><a class="reference internal" href="benchmarking/index.html">Benchmarking</a><details><summary><span class="toctree-toggle" role="presentation"><i class="fa-solid fa-chevron-down"></i></span></summary><ul>
<li class="toctree-l2"><a class="reference internal" href="benchmarking/environment.html">Environment</a></li>
<li class="toctree-l2"><a class="reference internal" href="benchmarking/ballista.html">Ballista</a></li>
@@ -366,7 +367,7 @@
<i class="fa-solid fa-home"></i>
</a>
</li>
- <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">Ballista Development</span></li>
+ <li class="breadcrumb-item active" aria-current="page"><span class="ellipsis">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</span></li>
</ul>
</nav>
</div>
@@ -401,201 +402,203 @@
specific language governing permissions and limitations
under the License.
-->
-<section id="ballista-development">
-<h1>Ballista Development<a class="headerlink" href="#ballista-development" title="Link to this heading">#</a></h1>
-<p>We welcome participation from everyone and encourage you to join us, ask
-questions, and get involved.</p>
-<p>All participation in the Apache DataFusion Ballista project is governed by the
-Apache Software Foundation’s <a class="reference external" href="https://www.apache.org/foundation/policies/conduct.html">code of
-conduct</a>.</p>
-<section id="development-environment">
-<h2>Development Environment<a class="headerlink" href="#development-environment" title="Link to this heading">#</a></h2>
-<p>The easiest way to get started if you are using VSCode or IntelliJ IDEA is to open the provided <a class="reference external" href="https://containers.dev/overview">Dev Container</a>
-which will install all the required dependencies including Rust, <code class="docutils literal notranslate"><span class="pre">protoc</span></code>, Docker, and Node.js. A Dev Container is a
-development environment that runs in a Docker container. It is configured with all the required dependencies to
-build and test the project. Other supporting tools that use Dev Containers can be seen
-<a class="reference external" href="https://containers.dev/supporting">here</a></p>
-<p>To use the Dev Container, open the project in VS Code and then click the “Reopen in Container” button in the
-bottom right corner of the IDE.</p>
-<p>If you are not using the Dev Container or VScode, you will need to install these dependencies yourself.</p>
-<ul class="simple">
-<li><p><a class="reference external" href="https://www.rust-lang.org/tools/install">Rust</a></p></li>
-<li><p><a class="reference external" href="https://protobuf.dev/downloads/">Protobuf Compiler</a> is required to build the project.</p></li>
-<li><p><a class="reference external" href="https://docs.docker.com/get-docker/">Docker</a> is required to run the integration tests.</p></li>
-<li><p><a class="reference external" href="https://nodejs.org/en/download/">Node.js</a> is only needed to run <code class="docutils literal notranslate"><span class="pre">npx</span> <span class="pre">prettier</span></code> over the Markdown docs.</p></li>
-<li><p>The <code class="docutils literal notranslate"><span class="pre">wasm32-unknown-unknown</span></code> target and <a class="reference external" href="https://trunk-rs.github.io/trunk/">Trunk</a> are only needed
-to build the Web TUI; see the <a class="reference internal" href="../user-guide/cli.html#running-the-web-tui-locally"><span class="std std-ref">Ballista CLI guide</span></a>.</p></li>
-</ul>
+<section id="dataframe-cache-design-dataframe-cache">
+<h1>DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)<a class="headerlink" href="#dataframe-cache-design-dataframe-cache" title="Link to this heading">#</a></h1>
+<section id="goal">
+<h2>Goal<a class="headerlink" href="#goal" title="Link to this heading">#</a></h2>
+<p><code class="docutils literal notranslate"><span class="pre">df.cache()</span></code> marks a DataFrame to be <strong>computed once and reused</strong> by later
+actions instead of recomputed. In Ballista a cached dataset is <strong>shuffle output
+that is kept (pinned) past normal job cleanup, plus a scheduler-side record of
+where that output lives</strong>. It reuses the native <a class="reference internal" href="shuffle.html"><span class="std std-doc">blocking shuffle</span></a>
+as its storage and transport — no separate cache store, no new wire format, and
+the read-back is an ordinary <code class="docutils literal notranslate"><span class="pre">ShuffleReaderExec</span></code>.</p>
+<p>The cache is a <strong>hint, not a correctness dependency</strong>: a cached node keeps its
+input subplan (lineage), so lost cache data is simply recomputed — never a wrong
+or missing answer.</p>
</section>
-<section id="build-the-project">
-<h2>Build the project<a class="headerlink" href="#build-the-project" title="Link to this heading">#</a></h2>
-<p>From the root of the project, build release binaries.</p>
-<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>cargo<span class="w"> </span>build<span class="w"> </span>--release
-</pre></div>
-</div>
+<section id="the-model">
+<h2>The model<a class="headerlink" href="#the-model" title="Link to this heading">#</a></h2>
+<p>In the <strong>logical plan</strong>, a cached dataset is a <code class="docutils literal notranslate"><span class="pre">BallistaCacheNode</span></code> wrapping the
+subplan to cache. Materialized, it has two pieces:</p>
+<ol class="arabic simple">
+<li><p><strong>Pinned shuffle output</strong> — the cached subplan runs as its own job; its final
+stage’s shuffle files (<code class="docutils literal notranslate"><span class="pre">{work_dir}/{job_id}/{stage_id}/…</span></code>) survive job cleanup.</p></li>
+<li><p><strong>A cross-job registry</strong> — scheduler state (the <code class="docutils literal notranslate"><span class="pre">CacheState</span></code> trait, in-memory
+<code class="docutils literal notranslate"><span class="pre">InMemoryCacheState</span></code> over <code class="docutils literal notranslate"><span class="pre">CacheRegistry</span></code>) mapping a cache key to the schema
+and <code class="docutils literal notranslate"><span class="pre">PartitionLocation</span></code>s of that output, so a <em>different</em> job can read it.</p></li>
+</ol>
+<p>A registry is needed because Ballista materializes the cache in a separate
+distributed <strong>job</strong>: the mapping from a cache to its shuffle output must outlive
+that job and be visible to other jobs, so the scheduler owns it explicitly.</p>
+<p><strong>Keying:</strong> <code class="docutils literal notranslate"><span class="pre">(session_id,</span> <span class="pre">cache_id)</span></code>, where <code class="docutils literal notranslate"><span class="pre">cache_id</span></code> is a per-<code class="docutils literal notranslate"><span class="pre">cache()</span></code>-call
+UUID in the <code class="docutils literal notranslate"><span class="pre">BallistaCacheNode</span></code>. Reuse happens whenever a submitted plan still
+contains that node (same <code class="docutils literal notranslate"><span class="pre">cache_id</span></code>) — the cached DataFrame reused directly or
+built upon. Dedup across <em>different</em> plans with equal content is deferred.</p>
</section>
-<section id="testing-the-project">
-<h2>Testing the project<a class="headerlink" href="#testing-the-project" title="Link to this heading">#</a></h2>
-<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>cargo<span class="w"> </span><span class="nb">test</span>
+<section id="end-to-end-flow">
+<h2>End-to-end flow<a class="headerlink" href="#end-to-end-flow" title="Link to this heading">#</a></h2>
+<p>Three roles: the <strong>client</strong> holds the plan, the <strong>scheduler</strong> owns planning and
+the registry, and the <strong>executors</strong> run tasks and hold the shuffle files.</p>
+<p><code class="docutils literal notranslate"><span class="pre">df.cache()</span></code> is lazy: it returns a DataFrame whose <strong>logical plan</strong> is wrapped in
+a <code class="docutils literal notranslate"><span class="pre">BallistaCacheNode</span></code> with a fresh <code class="docutils literal notranslate"><span class="pre">cache_id</span></code> — nothing executes yet. Reuse is
+keyed on that <code class="docutils literal notranslate"><span class="pre">cache_id</span></code>: any plan later submitted that still contains the node —
+the same cached DataFrame, or a query built on top of it — resolves to the same
+<strong>cache entry</strong> (the registry’s record of the materialized output). Calling
+<code class="docutils literal notranslate"><span class="pre">df.cache()</span></code> again mints a new <code class="docutils literal notranslate"><span class="pre">cache_id</span></code>.</p>
+<p>Miss and hit are not separate stories — they <strong>converge</strong>: a miss materializes
+and then reads the pinned output, while a hit does the same read and skips the
+materialize.</p>
+<div class="highlight-text notranslate"><div class="highlight"><pre><span></span> df.cache() client; lazy — wraps the logical
+ │ plan in BallistaCacheNode
+ ▼ {cache_id, session_id, input}
+ submit a plan carrying the node client ──► scheduler
+ │
+ ▼
+ lookup((session_id, cache_id)) scheduler
+ │
+ ┌────────────┴─────────────┐
+ MISS HIT
+ │ │
+ ▼ ▼
+ begin_materialization validate locations
+ run `input` as its own against live executors
+ job (two-job split, #1993) (any dead ⇒ re-MISS)
+ → executors write the │
+ pinned shuffle files │
+ │ │
+ ▼ │
+ complete_materialization │
+ → Materialized (fresh locs) │
+ │ │
+ └────────────┬──────────────┘
+ ▼
+ lower BallistaCacheNode ──► physical ShuffleReaderExec,
+ reading the pinned output served by the executors
+ │
+ ▼
+ results ──► client
</pre></div>
</div>
-<section id="tpc-h-plan-stability-golden-files">
-<h3>TPC-H plan-stability golden files<a class="headerlink" href="#tpc-h-plan-stability-golden-files" title="Link to this heading">#</a></h3>
-<p>The <code class="docutils literal notranslate"><span class="pre">tpch_plan_stability</span></code> test suite in the <code class="docutils literal notranslate"><span class="pre">ballista-scheduler</span></code> crate guards the shape of the
-distributed physical plans that Ballista produces for the 22 TPC-H queries. For each query it plans
-the SQL and compares the staged plan text — the stage boundaries (<code class="docutils literal notranslate"><span class="pre">===</span> <span class="pre">Stage</span> <span class="pre">N</span> <span class="pre">===</span></code> banners),
-<code class="docutils literal notranslate"><span class="pre">ShuffleWriterExec</span></code> / <code class="docutils literal notranslate"><span class="pre">UnresolvedShuffleExec</span></code> nodes, and any exchange reuse — against an approved
-“golden” file. The queries are planned against a synthetic statistics-only table, so the suite is
-deterministic and runs without TPC-H data or a running cluster.</p>
-<p>The golden files live in
-<a class="reference external" href="https://github.com/apache/datafusion-ballista/tree/main/ballista/scheduler/tests/tpch_plan_stability/approved"><code class="docutils literal notranslate"><span class="pre">ballista/scheduler/tests/tpch_plan_stability/approved/</span></code></a>
-(<code class="docutils literal notranslate"><span class="pre">q1.txt</span></code> through <code class="docutils literal notranslate"><span class="pre">q22.txt</span></code>). When a change alters distributed planning, the suite fails with a
-plan-drift diff. Review the diff: if the change is unintended, fix the regression; if the new plans
-are correct, regenerate the golden files:</p>
-<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>./dev/update-tpch-plan-stability.sh
-</pre></div>
-</div>
-<p>This runs the suite in generate mode
-(<code class="docutils literal notranslate"><span class="pre">BALLISTA_GENERATE_GOLDEN=1</span> <span class="pre">cargo</span> <span class="pre">test</span> <span class="pre">-p</span> <span class="pre">ballista-scheduler</span> <span class="pre">--test</span> <span class="pre">tpch_plan_stability</span></code>) and
-rewrites the approved files. Review the resulting changes under
-<code class="docutils literal notranslate"><span class="pre">ballista/scheduler/tests/tpch_plan_stability/approved/</span></code> and commit them together with the code
-change that caused them.</p>
+<p><strong>Cold path — first action (miss → materialize):</strong></p>
+<ol class="arabic simple">
+<li><p>The client submits a plan containing <code class="docutils literal notranslate"><span class="pre">BallistaCacheNode{cache_id,</span> <span class="pre">session_id,</span> <span class="pre">input}</span></code>.</p></li>
+<li><p>The scheduler forms <code class="docutils literal notranslate"><span class="pre">key</span> <span class="pre">=</span> <span class="pre">(session_id,</span> <span class="pre">cache_id)</span></code> and calls <code class="docutils literal notranslate"><span class="pre">lookup(key)</span></code> → miss.</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">begin_materialization(key,</span> <span class="pre">job)</span></code> claims the key; the scheduler runs <code class="docutils literal notranslate"><span class="pre">input</span></code>
+as its <strong>own job</strong> (a logical-plan rewrite that splits the query into two
+jobs, see #1993), writing pinned shuffle output on the executors.</p></li>
+<li><p>On that job’s <code class="docutils literal notranslate"><span class="pre">JobFinished</span></code>, <code class="docutils literal notranslate"><span class="pre">complete_materialization(key,</span> <span class="pre">job,</span> <span class="pre">schema,</span> <span class="pre">locations)</span></code> records the output → the cache entry becomes <code class="docutils literal notranslate"><span class="pre">Materialized</span></code>.</p></li>
+<li><p>The outer query reads that output and returns results — so the first action
+both materializes and reads back.</p></li>
+</ol>
+<p><strong>Warm path — later action (hit → read-back):</strong></p>
+<ol class="arabic simple">
+<li><p>The client submits a plan that still carries the same <code class="docutils literal notranslate"><span class="pre">cache_id</span></code>.</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">lookup(key)</span></code> → hit. Its locations are validated against the live-executor
+set; a dead location downgrades the hit to a miss (see <a class="reference internal" href="#invalidation"><span class="xref myst">Invalidation</span></a>).</p></li>
+<li><p>The scheduler lowers the <code class="docutils literal notranslate"><span class="pre">BallistaCacheNode</span></code> to a <strong>physical</strong>
+<code class="docutils literal notranslate"><span class="pre">ShuffleReaderExec</span></code> over the recorded locations — the same mechanism stages
+use to read each other — so <code class="docutils literal notranslate"><span class="pre">input</span></code> is never planned or executed.</p></li>
+<li><p>Results return with no materialization job.</p></li>
+</ol>
+<p><strong>Ordering & concurrency.</strong> Within one action the read is sequenced after the
+create: the read’s <code class="docutils literal notranslate"><span class="pre">ShuffleReaderExec</span></code> needs the recorded locations, so the
+scheduler resolves it only on the create job’s <code class="docutils literal notranslate"><span class="pre">JobFinished</span></code> — event-driven, no
+polling. A <em>concurrent</em> action that finds the key
+<code class="docutils literal notranslate"><span class="pre">Pending</span></code> (<code class="docutils literal notranslate"><span class="pre">begin_materialization</span></code> → <code class="docutils literal notranslate"><span class="pre">AlreadyPending</span></code>) must either wait for that
+materialization or fall back to computing the subplan inline; the exact policy —
+wait-and-notify vs. poll vs. inline, and releasing waiters if the job aborts — is
+an open question.</p>
</section>
-</section>
-<section id="running-the-examples">
-<h2>Running the examples<a class="headerlink" href="#running-the-examples" title="Link to this heading">#</a></h2>
-<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span><span class="nb">cd</span><span class="w"> </span>examples
-cargo<span class="w"> </span>run<span class="w"> </span>--example<span class="w"> </span>standalone_sql<span class="w"> </span>--features<span class="o">=</span>ballista/standalone
+<section id="registry-lifecycle">
+<h2>Registry lifecycle<a class="headerlink" href="#registry-lifecycle" title="Link to this heading">#</a></h2>
+<div class="highlight-text notranslate"><div class="highlight"><pre><span></span> (absent) --begin--> Pending{job} --complete(job)--> Materialized
+ Pending{job} --abort(job)-----> (absent)
+ Materialized --invalidate------> (absent)
</pre></div>
</div>
-</section>
-<section id="building-the-python-client-from-source">
-<h2>Building the Python Client from Source<a class="headerlink" href="#building-the-python-client-from-source" title="Link to this heading">#</a></h2>
-<p>The Python client (<code class="docutils literal notranslate"><span class="pre">ballista</span></code> on PyPI) lives in the <code class="docutils literal notranslate"><span class="pre">python/</span></code> directory and is versioned and released
-independently from the main Ballista project, so it is intentionally not part of the default Cargo
-workspace. Building it from source uses <a class="reference external" href="https://www.maturin.rs/">maturin</a> to compile the Rust
-extension module and install it into a Python virtual environment.</p>
-<section id="prerequisites">
-<h3>Prerequisites<a class="headerlink" href="#prerequisites" title="Link to this heading">#</a></h3>
+<p>The contract is safe against the out-of-order events a distributed scheduler
+produces (a completion callback racing invalidation, teardown, or a re-claim):</p>
<ul class="simple">
-<li><p>Python 3.10 or newer</p></li>
-<li><p>The Rust toolchain (see <a class="reference internal" href="#development-environment"><span class="xref myst">Development Environment</span></a>)</p></li>
-<li><p><a class="reference external" href="https://protobuf.dev/downloads/"><code class="docutils literal notranslate"><span class="pre">protoc</span></code></a> on your <code class="docutils literal notranslate"><span class="pre">PATH</span></code></p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">begin_materialization</span></code> atomically gives one job the claim.</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">complete_materialization</span></code> promotes <strong>only</strong> a <code class="docutils literal notranslate"><span class="pre">Pending</span></code> entry owned by that
+job (duplicate completions are idempotent); anything else is <code class="docutils literal notranslate"><span class="pre">Stale</span></code>, so a
+late completion cannot resurrect a removed entry or overwrite a newer claim.</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">abort_materialization</span></code> clears a failed job’s pending claim.</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">invalidate</span></code> / <code class="docutils literal notranslate"><span class="pre">invalidate_executor</span></code> / <code class="docutils literal notranslate"><span class="pre">remove_session</span></code> remove entries and
+<strong>return them</strong> so the caller can unpin the shuffle files (Ballista deletes
+explicitly — there is no reclaimer).</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">pinned_stages()</span></code> lists the <code class="docutils literal notranslate"><span class="pre">(job_id,</span> <span class="pre">stage_id)</span></code> stages to protect from cleanup.</p></li>
</ul>
-<p>All commands in this section are run from the <code class="docutils literal notranslate"><span class="pre">python/</span></code> directory:</p>
-<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span><span class="nb">cd</span><span class="w"> </span>python
-</pre></div>
-</div>
</section>
-<section id="create-a-virtual-environment">
-<h3>Create a virtual environment<a class="headerlink" href="#create-a-virtual-environment" title="Link to this heading">#</a></h3>
-<p>Using <code class="docutils literal notranslate"><span class="pre">pip</span></code>:</p>
-<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>python3<span class="w"> </span>-m<span class="w"> </span>venv<span class="w"> </span>.venv
-<span class="nb">source</span><span class="w"> </span>.venv/bin/activate
-pip3<span class="w"> </span>install<span class="w"> </span>--upgrade<span class="w"> </span>pip
-pip3<span class="w"> </span>install<span class="w"> </span>-r<span class="w"> </span>requirements.txt
-</pre></div>
-</div>
-<p>The <code class="docutils literal notranslate"><span class="pre">pip</span></code> upgrade is required because <code class="docutils literal notranslate"><span class="pre">maturin</span> <span class="pre">develop</span></code> invokes <code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">--group</span></code>, which
-needs pip 25.1 or newer. The pip shipped with <code class="docutils literal notranslate"><span class="pre">python3</span> <span class="pre">-m</span> <span class="pre">venv</span></code> is often older than that.</p>
-<p>Using <a class="reference external" href="https://docs.astral.sh/uv/"><code class="docutils literal notranslate"><span class="pre">uv</span></code></a>:</p>
-<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>uv<span class="w"> </span>sync<span class="w"> </span>--dev<span class="w"> </span>--no-install-package<span class="w"> </span>ballista
-</pre></div>
-</div>
+<section id="pinning">
+<h2>Pinning<a class="headerlink" href="#pinning" title="Link to this heading">#</a></h2>
+<p>Shuffle output is normally reclaimed after a job finishes. A cached stage must
+be exempted, so job-data cleanup consults <code class="docutils literal notranslate"><span class="pre">pinned_stages()</span></code> and skips those
+stage dirs. Pins are per-<code class="docutils literal notranslate"><span class="pre">(job_id,</span> <span class="pre">stage_id)</span></code> because shuffle data is stored and
+deleted per stage, and a cached stage is often an <em>intermediate</em> stage (reaped
+immediately on job success) that a job-level pin could not protect.</p>
</section>
-<section id="build-and-install">
-<h3>Build and install<a class="headerlink" href="#build-and-install" title="Link to this heading">#</a></h3>
-<p><code class="docutils literal notranslate"><span class="pre">maturin</span> <span class="pre">develop</span></code> compiles the Rust extension and installs it into the active virtual environment
-as an editable package, so subsequent Python imports pick up your local changes.</p>
-<p>Using <code class="docutils literal notranslate"><span class="pre">pip</span></code>:</p>
-<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>maturin<span class="w"> </span>develop<span class="w"> </span><span class="c1"># debug build</span>
-maturin<span class="w"> </span>develop<span class="w"> </span>--release<span class="w"> </span><span class="c1"># release build (slower to compile, faster at runtime)</span>
-</pre></div>
-</div>
-<p>Using <code class="docutils literal notranslate"><span class="pre">uv</span></code>:</p>
-<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>uv<span class="w"> </span>run<span class="w"> </span>--no-project<span class="w"> </span>maturin<span class="w"> </span>develop<span class="w"> </span>--uv
-</pre></div>
-</div>
-<p>To produce a wheel without installing it (for example, to share or publish):</p>
-<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>uv<span class="w"> </span>run<span class="w"> </span>--no-project<span class="w"> </span>maturin<span class="w"> </span>build<span class="w"> </span>--release<span class="w"> </span>--strip
-</pre></div>
-</div>
-</section>
-<section id="run-the-tests">
-<h3>Run the tests<a class="headerlink" href="#run-the-tests" title="Link to this heading">#</a></h3>
-<p>Using <code class="docutils literal notranslate"><span class="pre">pip</span></code>:</p>
-<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>python3<span class="w"> </span>-m<span class="w"> </span>pytest
-</pre></div>
-</div>
-<p>Using <code class="docutils literal notranslate"><span class="pre">uv</span></code>:</p>
-<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>uv<span class="w"> </span>run<span class="w"> </span>--no-project<span class="w"> </span>pytest
-</pre></div>
-</div>
-<p>For more detail on the underlying workflow, including tips for improving build speed, see the
-<a class="reference external" href="https://datafusion.apache.org/python/contributor-guide/introduction.html">DataFusion Python contributor guide</a>.</p>
+<section id="invalidation">
+<h2>Invalidation<a class="headerlink" href="#invalidation" title="Link to this heading">#</a></h2>
+<p>An entry is removed when its data is no longer trustworthy or needed:</p>
+<ul class="simple">
+<li><p><strong>Executor loss</strong> (<code class="docutils literal notranslate"><span class="pre">invalidate_executor</span></code>) — any materialized entry with a
+partition on the lost executor is dropped.</p></li>
+<li><p><strong>Session teardown</strong> (<code class="docutils literal notranslate"><span class="pre">remove_session</span></code>) — all of a session’s entries are
+dropped.</p></li>
+<li><p><strong>Explicit / eviction</strong> (<code class="docutils literal notranslate"><span class="pre">invalidate</span></code>) — a single entry is dropped (used by
+the future eviction sweeper).</p></li>
+</ul>
+<p>Invalidation is <strong>whole-entry</strong>: the entire cache is dropped and re-materialized
+in full on the next access — we never serve a cache with missing partitions
+(per-partition rebuild is a future optimization). Because a cached node keeps its
+lineage, invalidation always degrades to a recompute, never a wrong answer.</p>
+<p>Each removal <strong>returns the removed entries</strong> so the caller can unpin and delete
+their shuffle files — Ballista has no background reclaimer, so cleanup is
+explicit. A removed entry’s next <code class="docutils literal notranslate"><span class="pre">lookup</span></code> is a miss, which re-triggers
+materialization.</p>
+<blockquote>
+<div><p><strong>Executor loss during materialization (executor-loss follow-up).</strong> If an
+executor is lost after the cached stage finishes but before its job completes,
+<code class="docutils literal notranslate"><span class="pre">complete_materialization</span></code> can record a now-dead location (the in-graph
+recovery only rebuilds output an unfinished stage still reads). The fix is
+<strong>validate-on-read</strong>: on a hit the planner checks locations against the
+live-executor set and, if any is dead, treats it as a miss and <code class="docutils literal notranslate"><span class="pre">invalidate</span></code>s
+it — enforcing “never read from a dead executor” under all timings.</p>
+</div></blockquote>
</section>
+<section id="durability-reconciliation">
+<h2>Durability & reconciliation<a class="headerlink" href="#durability-reconciliation" title="Link to this heading">#</a></h2>
+<p><code class="docutils literal notranslate"><span class="pre">CacheState</span></code> is a pluggable backend; the in-memory one loses entries on scheduler
+restart (caches become orphaned files, reclaimed by cleanup), and a durable
+backend can persist and rehydrate them — an optional efficiency extension. A
+stale or missing location is caught by validate-on-read (executor gone) or the
+<code class="docutils literal notranslate"><span class="pre">ShuffleReader</span></code> fetch-failure path (file gone) and degrades to re-materialization.</p>
</section>
-<section id="benchmarking">
-<h2>Benchmarking<a class="headerlink" href="#benchmarking" title="Link to this heading">#</a></h2>
-<p>For performance testing and benchmarking with TPC-H and other datasets, see the <a class="reference external" href="https://github.com/apache/datafusion-ballista/blob/main/benchmarks/README.md">benchmarks README</a>.</p>
-<p>This includes instructions for:</p>
+<section id="future-work">
+<h2>Future work<a class="headerlink" href="#future-work" title="Link to this heading">#</a></h2>
<ul class="simple">
-<li><p>Generating TPC-H test data</p></li>
-<li><p>Running benchmarks against DataFusion and Ballista</p></li>
-<li><p>Comparing performance with Apache Spark</p></li>
-<li><p>Running load tests</p></li>
+<li><p><strong>Eviction:</strong> entries today leave only via invalidation/executor-loss/session
+teardown, so a long-running scheduler accumulates cache data. Planned: a TTL /
+size-cap sweeper (via <code class="docutils literal notranslate"><span class="pre">invalidate</span></code>) and/or idle-session expiry (via
+<code class="docutils literal notranslate"><span class="pre">remove_session</span></code>), using the removal-returns-entries contract above.</p></li>
+<li><p><strong>Locality:</strong> place read-back tasks on the executor holding each partition
+(<code class="docutils literal notranslate"><span class="pre">TaskDistributionPolicy::Custom</span></code>), Flight fetch as fallback (see <a class="reference external" href="https://github.com/apache/datafusion-ballista/issues/2319">#2319</a>).</p></li>
</ul>
</section>
-<section id="datafusion-dependency">
-<h2>DataFusion dependency<a class="headerlink" href="#datafusion-dependency" title="Link to this heading">#</a></h2>
-<p>Between releases, <code class="docutils literal notranslate"><span class="pre">main</span></code> depends on a pinned git revision of
-<a class="reference external" href="https://github.com/apache/datafusion"><code class="docutils literal notranslate"><span class="pre">apache/datafusion</span></code></a> <code class="docutils literal notranslate"><span class="pre">main</span></code> rather than a crates.io release, so
-that breaking changes arrive in small increments instead of piling up at release time.</p>
-<p>The <code class="docutils literal notranslate"><span class="pre">datafusion*</span></code> entries in the workspace <code class="docutils literal notranslate"><span class="pre">Cargo.toml</span></code> move through one cycle per release:</p>
-<div class="pst-scrollable-table-container"><table class="table">
-<thead>
-<tr class="row-odd"><th class="head"><p>Phase</p></th>
-<th class="head"><p><code class="docutils literal notranslate"><span class="pre">datafusion*</span></code> dependency</p></th>
-</tr>
-</thead>
-<tbody>
-<tr class="row-even"><td><p>Between releases</p></td>
-<td><p>pinned git <code class="docutils literal notranslate"><span class="pre">rev</span></code> of DataFusion <code class="docutils literal notranslate"><span class="pre">main</span></code></p></td>
-</tr>
-<tr class="row-odd"><td><p>DataFusion release candidate</p></td>
-<td><p>the RC tag, to validate early</p></td>
-</tr>
-<tr class="row-even"><td><p>DataFusion released</p></td>
-<td><p>the crates.io version</p></td>
-</tr>
-<tr class="row-odd"><td><p>Ballista release branch cut</p></td>
-<td><p>(branch keeps the crates.io version)</p></td>
-</tr>
-<tr class="row-even"><td><p>Back on <code class="docutils literal notranslate"><span class="pre">main</span></code></p></td>
-<td><p>pinned git <code class="docutils literal notranslate"><span class="pre">rev</span></code> of DataFusion <code class="docutils literal notranslate"><span class="pre">main</span></code> again</p></td>
-</tr>
-</tbody>
-</table>
-</div>
-<p>A release branch therefore only ever depends on published crates. Each <code class="docutils literal notranslate"><span class="pre">rev</span></code> bump on <code class="docutils literal notranslate"><span class="pre">main</span></code> is a PR
-that ports any DataFusion API changes, refreshes affected golden files, and keeps <code class="docutils literal notranslate"><span class="pre">cargo</span> <span class="pre">test</span></code> and
-<code class="docutils literal notranslate"><span class="pre">cargo</span> <span class="pre">clippy</span> <span class="pre">--all-targets</span> <span class="pre">--workspace</span> <span class="pre">--</span> <span class="pre">-D</span> <span class="pre">warnings</span></code> green.</p>
-<p>The Python client in <code class="docutils literal notranslate"><span class="pre">python/</span></code> is a separate Cargo workspace that also depends on
-<a class="reference external" href="https://github.com/apache/datafusion-python">datafusion-python</a>. It can only move to a new
-DataFusion major version once datafusion-python has published one, usually some weeks after
-DataFusion, so it is released separately from the Rust crates. See the
-<a class="reference external" href="https://github.com/apache/datafusion-ballista/blob/main/python/dev/release/README.md">Python client release process</a>.</p>
-<p>The remaining release steps are described in the
-<a class="reference external" href="https://github.com/apache/datafusion-ballista/blob/main/dev/release/README.md">release process</a>.</p>
-</section>
-<section id="upgrade-notes">
-<h2>Upgrade notes<a class="headerlink" href="#upgrade-notes" title="Link to this heading">#</a></h2>
-<p>When a change breaks a public API, renames or removes a configuration key or CLI
-flag, changes a default that alters query plans or results, or changes the
-wire/serialization format, add an entry describing the change and the required
-action to the current unreleased page under
-<a class="reference internal" href="../upgrading/index.html"><span class="std std-doc"><code class="docutils literal notranslate"><span class="pre">docs/source/upgrading/</span></code></span></a>. This keeps the upgrade guide
-complete for the next release.</p>
+<section id="implementation-plan">
+<h2>Implementation plan<a class="headerlink" href="#implementation-plan" title="Link to this heading">#</a></h2>
+<ol class="arabic simple">
+<li><p><strong>Registry</strong> (done) — <code class="docutils literal notranslate"><span class="pre">CacheRegistry</span></code> + <code class="docutils literal notranslate"><span class="pre">CacheState</span></code>/<code class="docutils literal notranslate"><span class="pre">InMemoryCacheState</span></code>.</p></li>
+<li><p><strong>Materialize-on-miss + read-back</strong> — wire <code class="docutils literal notranslate"><span class="pre">lookup</span></code>/<code class="docutils literal notranslate"><span class="pre">begin</span></code>/<code class="docutils literal notranslate"><span class="pre">complete</span></code>/<code class="docutils literal notranslate"><span class="pre">abort</span></code>
+into planning and the job handlers; builds on the two-job rewrite in <a class="reference external" href="https://github.com/apache/datafusion-ballista/pull/1993">#1993</a>.</p></li>
+<li><p><strong>Pinning</strong> — teach job-data cleanup to honor <code class="docutils literal notranslate"><span class="pre">pinned_stages()</span></code>.</p></li>
+<li><p><strong>Locality</strong> — place read-back tasks where partitions live (<a class="reference external" href="https://github.com/apache/datafusion-ballista/issues/2319">#2319</a>).</p></li>
+<li><p><strong>Executor-loss invalidation + session teardown</strong> — wire
+<code class="docutils literal notranslate"><span class="pre">invalidate_executor</span></code> / <code class="docutils literal notranslate"><span class="pre">remove_session</span></code> and the unpin path, add
+validate-on-read on the hit path, and enable an end-to-end no-recompute test.</p></li>
+<li><p><strong>Eviction</strong> — TTL / size cap / idle-session expiry.</p></li>
+</ol>
</section>
</section>
@@ -610,20 +613,20 @@ complete for the next release.</p>
<div class="prev-next-area">
<a class="left-prev"
- href="user-personas.html"
+ href="shuffle.html"
title="previous page">
<i class="fa-solid fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">User Personas</p>
+ <p class="prev-next-title">Shuffle Design</p>
</div>
</a>
<a class="right-next"
- href="benchmarking/index.html"
+ href="code-organization.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Benchmarking</p>
+ <p class="prev-next-title">Ballista Code Organization</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
@@ -646,23 +649,15 @@ complete for the next release.</p>
</div>
<nav id="pst-page-toc-nav" class="page-toc" aria-labelledby="pst-page-navigation-heading-2">
<ul class="pst-show_toc_level nav section-nav flex-column">
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#development-environment">Development Environment</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#build-the-project">Build the project</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#testing-the-project">Testing the project</a><ul class="nav section-nav flex-column">
-<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#tpc-h-plan-stability-golden-files">TPC-H plan-stability golden files</a></li>
-</ul>
-</li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#running-the-examples">Running the examples</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#building-the-python-client-from-source">Building the Python Client from Source</a><ul class="nav section-nav flex-column">
-<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#prerequisites">Prerequisites</a></li>
-<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#create-a-virtual-environment">Create a virtual environment</a></li>
-<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#build-and-install">Build and install</a></li>
-<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#run-the-tests">Run the tests</a></li>
-</ul>
-</li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#benchmarking">Benchmarking</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion-dependency">DataFusion dependency</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#upgrade-notes">Upgrade notes</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#goal">Goal</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-model">The model</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#end-to-end-flow">End-to-end flow</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#registry-lifecycle">Registry lifecycle</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#pinning">Pinning</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#invalidation">Invalidation</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#durability-reconciliation">Durability & reconciliation</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#future-work">Future work</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#implementation-plan">Implementation plan</a></li>
</ul>
</nav></div>
@@ -670,7 +665,7 @@ complete for the next release.</p>
<div class="tocsection editthispage">
- <a href="https://github.com/apache/datafusion-ballista/edit/main/docs/source/contributors-guide/development.md">
+ <a href="https://github.com/apache/datafusion-ballista/edit/main/docs/source/contributors-guide/dataframe-cache.md">
<i class="fa-solid fa-pencil"></i>
@@ -685,7 +680,7 @@ complete for the next release.</p>
<div class="sidebar-secondary-item">
<div class="tocsection sourcelink">
- <a href="../_sources/contributors-guide/development.md.txt">
+ <a href="../_sources/contributors-guide/dataframe-cache.md.txt">
<i class="fa-solid fa-file-lines"></i> Show Source
</a>
</div>
diff --git a/contributors-guide/development.html b/contributors-guide/development.html
index 32e03c3b1..c70135af9 100644
--- a/contributors-guide/development.html
+++ b/contributors-guide/development.html
@@ -305,6 +305,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="user-personas.html">User Personas</a></li>
<li class="toctree-l1 current active"><a aria-current="page" class="current reference internal" href="#">Ballista Development</a></li>
diff --git a/contributors-guide/shuffle.html b/contributors-guide/shuffle.html
index 7ec99b776..4d7955da1 100644
--- a/contributors-guide/shuffle.html
+++ b/contributors-guide/shuffle.html
@@ -45,7 +45,7 @@
<script>DOCUMENTATION_OPTIONS.search_as_you_type = false;</script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
- <link rel="next" title="Ballista Code Organization" href="code-organization.html" />
+ <link rel="next" title="DataFrame Cache Design (DataFrame::cache())" href="dataframe-cache.html" />
<link rel="prev" title="Ballista Architecture" href="architecture.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="color-scheme" content="light dark"/>
@@ -305,6 +305,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1 current active"><a aria-current="page" class="current reference internal" href="#">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="development.html">Ballista Development</a></li>
@@ -701,11 +702,11 @@ properties above it preserves and which it trades away.</p>
</div>
</a>
<a class="right-next"
- href="code-organization.html"
+ href="dataframe-cache.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Ballista Code Organization</p>
+ <p class="prev-next-title">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
diff --git a/contributors-guide/user-personas.html b/contributors-guide/user-personas.html
index f4e1f3e86..5d6c9a46d 100644
--- a/contributors-guide/user-personas.html
+++ b/contributors-guide/user-personas.html
@@ -305,6 +305,7 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1 current active"><a aria-current="page" class="current reference internal" href="#">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="development.html">Ballista Development</a></li>
diff --git a/genindex.html b/genindex.html
index 5ac684bbe..59894f91f 100644
--- a/genindex.html
+++ b/genindex.html
@@ -300,6 +300,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="contributors-guide/development.html">Ballista Development</a></li>
diff --git a/index.html b/index.html
index 21efe885f..28ec133c5 100644
--- a/index.html
+++ b/index.html
@@ -306,6 +306,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="contributors-guide/development.html">Ballista Development</a></li>
@@ -415,6 +416,7 @@
<ul>
<li class="toctree-l1"><a class="reference internal" href="contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="contributors-guide/development.html">Ballista Development</a></li>
diff --git a/objects.inv b/objects.inv
index 64b40682d..645ae924f 100644
Binary files a/objects.inv and b/objects.inv differ
diff --git a/search.html b/search.html
index 86de9f356..0c9746cd7 100644
--- a/search.html
+++ b/search.html
@@ -299,6 +299,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="contributors-guide/development.html">Ballista Development</a></li>
diff --git a/searchindex.js b/searchindex.js
index 7a3e2bc27..4fb3b203f 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles":{"API changes":[[38,"api-changes"]],"AQE join behavior (opt-in)":[[37,"aqe-join-behavior-opt-in"]],"Adaptive Query Execution":[[61,"adaptive-query-execution"]],"Adaptive Query Execution (AQE)":[[25,"adaptive-query-execution-aqe"]],"Adding a persona":[[35,"adding-a-persona"]],"Apache DataFusion Ballista":[[36,null]],"Apache DataFusion Ballista 0.10.0 Changelog":[[0,null]],"Apache DataFusion Ballista 0.11.0 Changelog":[[1,null]],"Apache DataFusion Ballista 0.12 [...]
\ No newline at end of file
+Search.setIndex({"alltitles":{"API changes":[[39,"api-changes"]],"AQE join behavior (opt-in)":[[38,"aqe-join-behavior-opt-in"]],"Adaptive Query Execution":[[62,"adaptive-query-execution"]],"Adaptive Query Execution (AQE)":[[25,"adaptive-query-execution-aqe"]],"Adding a persona":[[36,"adding-a-persona"]],"Apache DataFusion Ballista":[[37,null]],"Apache DataFusion Ballista 0.10.0 Changelog":[[0,null]],"Apache DataFusion Ballista 0.11.0 Changelog":[[1,null]],"Apache DataFusion Ballista 0.12 [...]
\ No newline at end of file
diff --git a/upgrading/54.0.0.html b/upgrading/54.0.0.html
index eb9e922ad..2fb99de13 100644
--- a/upgrading/54.0.0.html
+++ b/upgrading/54.0.0.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/upgrading/55.0.0.html b/upgrading/55.0.0.html
index d45dba48d..daac0ac80 100644
--- a/upgrading/55.0.0.html
+++ b/upgrading/55.0.0.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/upgrading/index.html b/upgrading/index.html
index 2b2e4f6b6..4e4a55e3e 100644
--- a/upgrading/index.html
+++ b/upgrading/index.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/user-guide/cli.html b/user-guide/cli.html
index 89e89f315..7e2d76fe7 100644
--- a/user-guide/cli.html
+++ b/user-guide/cli.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/user-guide/configs.html b/user-guide/configs.html
index 39622a897..06b1f603c 100644
--- a/user-guide/configs.html
+++ b/user-guide/configs.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/user-guide/deployment/cargo-install.html b/user-guide/deployment/cargo-install.html
index 1523f7c2a..e2257659d 100644
--- a/user-guide/deployment/cargo-install.html
+++ b/user-guide/deployment/cargo-install.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/user-guide/deployment/docker-compose.html b/user-guide/deployment/docker-compose.html
index 2983c8bce..f3673b76a 100644
--- a/user-guide/deployment/docker-compose.html
+++ b/user-guide/deployment/docker-compose.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/user-guide/deployment/docker.html b/user-guide/deployment/docker.html
index c784017ed..c8b652462 100644
--- a/user-guide/deployment/docker.html
+++ b/user-guide/deployment/docker.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/user-guide/deployment/index.html b/user-guide/deployment/index.html
index bf0361782..1c8f2f7b1 100644
--- a/user-guide/deployment/index.html
+++ b/user-guide/deployment/index.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/user-guide/deployment/kubernetes.html b/user-guide/deployment/kubernetes.html
index 46473bc3d..d7271ef7e 100644
--- a/user-guide/deployment/kubernetes.html
+++ b/user-guide/deployment/kubernetes.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/user-guide/deployment/quick-start.html b/user-guide/deployment/quick-start.html
index b33310fe6..4fbb7aec9 100644
--- a/user-guide/deployment/quick-start.html
+++ b/user-guide/deployment/quick-start.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/user-guide/extending-components.html b/user-guide/extending-components.html
index a32978ded..39cc45a8a 100644
--- a/user-guide/extending-components.html
+++ b/user-guide/extending-components.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/user-guide/extensions-example.html b/user-guide/extensions-example.html
index 56d7fb2e2..a8a673b46 100644
--- a/user-guide/extensions-example.html
+++ b/user-guide/extensions-example.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/user-guide/faq.html b/user-guide/faq.html
index 96065f64d..8bbf27b41 100644
--- a/user-guide/faq.html
+++ b/user-guide/faq.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/user-guide/flightsql.html b/user-guide/flightsql.html
index 409044931..66d5adfc8 100644
--- a/user-guide/flightsql.html
+++ b/user-guide/flightsql.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/user-guide/history-server.html b/user-guide/history-server.html
index c1bd341c3..327db1a0a 100644
--- a/user-guide/history-server.html
+++ b/user-guide/history-server.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/user-guide/introduction.html b/user-guide/introduction.html
index 4a6345e52..ded265692 100644
--- a/user-guide/introduction.html
+++ b/user-guide/introduction.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/user-guide/metrics.html b/user-guide/metrics.html
index 43e4d0226..362d95c16 100644
--- a/user-guide/metrics.html
+++ b/user-guide/metrics.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/user-guide/python/cloud-storage.html b/user-guide/python/cloud-storage.html
index 8345c78f9..6366c1938 100644
--- a/user-guide/python/cloud-storage.html
+++ b/user-guide/python/cloud-storage.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/user-guide/python/index.html b/user-guide/python/index.html
index c6194c443..73742a7e7 100644
--- a/user-guide/python/index.html
+++ b/user-guide/python/index.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/user-guide/python/jupyter.html b/user-guide/python/jupyter.html
index 1cc327d0b..ebb891141 100644
--- a/user-guide/python/jupyter.html
+++ b/user-guide/python/jupyter.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/user-guide/python/quickstart.html b/user-guide/python/quickstart.html
index 02bbbd6f3..eb3401a19 100644
--- a/user-guide/python/quickstart.html
+++ b/user-guide/python/quickstart.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/user-guide/rust.html b/user-guide/rust.html
index d9edec457..9d0e3c566 100644
--- a/user-guide/rust.html
+++ b/user-guide/rust.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/user-guide/scheduler.html b/user-guide/scheduler.html
index 1e9ffa956..4235e3890 100644
--- a/user-guide/scheduler.html
+++ b/user-guide/scheduler.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
diff --git a/user-guide/tuning-guide.html b/user-guide/tuning-guide.html
index 9f0a66d02..53910869b 100644
--- a/user-guide/tuning-guide.html
+++ b/user-guide/tuning-guide.html
@@ -305,6 +305,7 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/architecture.html">Ballista Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/shuffle.html">Shuffle Design</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/dataframe-cache.html">DataFrame Cache Design (<code class="docutils literal notranslate"><span class="pre">DataFrame::cache()</span></code>)</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/code-organization.html">Ballista Code Organization</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/user-personas.html">User Personas</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors-guide/development.html">Ballista Development</a></li>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org