You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/01 00:48:56 UTC

(datafusion-comet) branch asf-site updated: Publish built docs triggered by ec66afe56b1b9eb5ecc695169feb26495917eedd

This is an automated email from the ASF dual-hosted git repository.

github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git


The following commit(s) were added to refs/heads/asf-site by this push:
     new 2d264292ee Publish built docs triggered by ec66afe56b1b9eb5ecc695169feb26495917eedd
2d264292ee is described below

commit 2d264292ee6329b45d805ad2c05a36ef0fbfea96
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Thu Oct 1 00:48:51 2026 +0000

    Publish built docs triggered by ec66afe56b1b9eb5ecc695169feb26495917eedd
---
 ...id-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg |  2 +-
 .../contributor-guide/memory_management.md.txt     |  4 +-
 _sources/contributor-guide/plugin_overview.md.txt  |  9 +++
 _sources/user-guide/latest/configs.md.txt          |  2 +-
 _sources/user-guide/latest/tuning/memory.md.txt    | 68 +++++++++++++++++++++-
 contributor-guide/memory_management.html           |  4 +-
 contributor-guide/plugin_overview.html             |  8 +++
 searchindex.js                                     |  2 +-
 user-guide/latest/configs.html                     |  2 +-
 user-guide/latest/tuning/memory.html               | 64 ++++++++++++++++++--
 10 files changed, 152 insertions(+), 13 deletions(-)

diff --git a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
index c16fe7ce1d..78c300f693 100644
--- a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
+++ b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/memory_management.md.txt b/_sources/contributor-guide/memory_management.md.txt
index a699768379..d899d02029 100644
--- a/_sources/contributor-guide/memory_management.md.txt
+++ b/_sources/contributor-guide/memory_management.md.txt
@@ -572,7 +572,9 @@ A checklist for triaging an executor OOM kill:
    them. A failed task with `SparkOutOfMemoryError` and a surviving executor is Spark's managed
    memory pool, which is the only one of the three that is recoverable at task level.
 2. Compare `allocated` against `reserved` in the executor's `Comet native memory usage` log lines
-   leading up to the kill, or `native_allocated` against `comet_memory_reserved_total` in a trace.
+   leading up to the kill, or `nativeAllocated` against `poolsReserved` in its
+   `CometExecutorMemoryUsage` events when the event log records them, or `native_allocated`
+   against `comet_memory_reserved_total` in a trace.
    A large excess points at undeclared native allocations; a small excess points at the budget
    simply being too small, or at the JVM side, which the same lines report as `JVM Arrow allocated`.
 3. Check `spark.comet.batchSize` against the schema width. Peak memory scales with
diff --git a/_sources/contributor-guide/plugin_overview.md.txt b/_sources/contributor-guide/plugin_overview.md.txt
index a97cc15f41..89586bf7b6 100644
--- a/_sources/contributor-guide/plugin_overview.md.txt
+++ b/_sources/contributor-guide/plugin_overview.md.txt
@@ -57,6 +57,15 @@ to size them.
 
 When the driver or an executor stops, the plugin shuts down Comet's native tokio runtime in that JVM.
 
+When `spark.eventLog.enabled` is `true`, the executor plugin also carries each sample of the executor's native memory
+usage log to the driver plugin. For each executor, the driver plugin keeps the sample with the most untracked memory
+and the last sample of every minute, and posts them to Spark's listener bus as `CometExecutorMemoryUsage` events so
+that the event log records them. A listener on a listener bus queue of its own posts them at the executor's first
+heartbeat a minute or more after the first of them arrived, which also ends an idle executor's minute, and posts what
+the driver plugin holds when an executor is removed and when the application ends, since an executor that goes away
+sends nothing more. See
+[Reading the Memory Usage Log from the Event Log](../user-guide/latest/tuning/memory.md#reading-the-memory-usage-log-from-the-event-log).
+
 `CometSparkSessionExtensions` can also be registered without the plugin, through `spark.sql.extensions` or
 `SparkSession.Builder.withExtensions`. Most of Comet's test suites and the Spark SQL tests enable Comet this way, so
 none of the driver plugin's steps run for them.
diff --git a/_sources/user-guide/latest/configs.md.txt b/_sources/user-guide/latest/configs.md.txt
index ace9ba4adb..b24e39793a 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -161,7 +161,7 @@ shuffle, including applications that rely only on remote reliable storage or dec
 | `spark.comet.exec.memoryPool` | The type of memory pool to be used for Comet native execution when running Spark in off-heap mode. Available pool types are `greedy_unified` and `fair_unified`. For more information, refer to the [Comet Tuning Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html). | fair_unified |
 | `spark.comet.exec.memoryPool.fraction` | Deprecated: this config will be removed in a future major release. It does not leave room in spark.memory.offHeap.size for native memory that Comet's memory pools do not track, because Spark hands out the whole off-heap pool whatever this is set to. Size spark.executor.memoryOverhead for that memory instead. Only applies to off-heap mode, where the fair_unified pool limits each memory consumer in a task to this fraction of the off-heap size divi [...]
 | `spark.comet.maxTempDirectorySize` | The maximum amount of data (in bytes) stored inside the temporary directories used by native operators when spilling. Applied to each Comet native plan separately, and a Spark task can run more than one native plan at a time, so an executor running N concurrent tasks may use more than N times this value on shared local disks. Once the limit is reached, further spills will fail and the query will error out. | 107374182400b |
-| `spark.comet.memory.logInterval` | How often each executor logs its native memory usage at INFO level while Comet native plans are running: the bytes the native allocator has handed out, the bytes reserved in Comet's memory pools, and the Arrow memory Comet holds on the JVM side. The difference between the first two is native memory that the pools are not accounting for. The executor logs one line per interval however many tasks are running, and one more after the last plan finishes. I [...]
+| `spark.comet.memory.logInterval` | How often each executor logs its native memory usage at INFO level while Comet native plans are running: the bytes the native allocator has handed out, the bytes reserved in Comet's memory pools, and the Arrow memory Comet holds on the JVM side. The difference between the first two is native memory that the pools are not accounting for. The executor logs one line per interval however many tasks are running, and one more after the last plan finishes. I [...]
 | `spark.comet.tracing.enabled` | Enable fine-grained tracing of events and memory usage. For more information, refer to the [Comet Tracing Guide](https://datafusion.apache.org/comet/contributor-guide/tracing.html). | false |
 <!-- prettier-ignore-end -->
 <!--END:CONFIG_TABLE-->
diff --git a/_sources/user-guide/latest/tuning/memory.md.txt b/_sources/user-guide/latest/tuning/memory.md.txt
index 9b38348c42..1f6ea98a2b 100644
--- a/_sources/user-guide/latest/tuning/memory.md.txt
+++ b/_sources/user-guide/latest/tuning/memory.md.txt
@@ -157,15 +157,19 @@ Comet native memory usage: allocated 5412.3 MiB, reserved 3890.0 MiB (16 native
   which allocator a buffer is charged to rather than where it was allocated, so treat the JVM's
   part as an estimate.
 
+When the application writes an event log and runs the Comet plugin, the event log also records a
+summary of these samples; see [Reading the Memory Usage Log from the Event Log].
+
 Comet's untracked memory is what Comet holds outside the JVM heap that Spark's off-heap pool does
 not account for: `allocated`, plus the JVM Arrow figure less the part imported from native, minus
 `reserved`. It is the part of Comet's footprint that has to fit in `spark.executor.memoryOverhead`,
 alongside the JVM's own non-heap memory. To size the overhead from it:
 
 1. Run a representative workload and find the line with the most untracked memory in each
-   executor's log. Take all the figures from the same line: they are sampled together, and figures
-   from different lines describe different moments. Setting `spark.comet.memory.logInterval=1s` for
-   this run makes a short-lived peak less likely to fall between samples.
+   executor's log, or the event with the most for each executor in the event log. Take all the
+   figures from the same line or event: they are sampled together, and figures from different
+   samples describe different moments. Setting `spark.comet.memory.logInterval=1s` for this run
+   makes a short-lived peak less likely to fall between samples.
 2. Start from the overhead the executors had before Comet was enabled, which covers the JVM's own
    non-heap memory, and add the most untracked memory seen on any executor.
 3. Add a margin on top. The log can miss the true peak between samples, and none of the figures
@@ -199,6 +203,64 @@ The executor logs one more line after its last native plan finishes, and an `all
 `spark.comet.memory.logInterval` is read when an executor starts its first Comet native plan, so set
 it when the application is submitted. Set it to `0` to turn the log off.
 
+[Reading the Memory Usage Log from the Event Log]: #reading-the-memory-usage-log-from-the-event-log
+
+### Reading the Memory Usage Log from the Event Log
+
+When `spark.eventLog.enabled` is `true` and the application runs the Comet plugin
+(`spark.plugins=org.apache.spark.CometPlugin`), each executor also sends its samples to the driver,
+which writes a summary of them to the application's event log. The event log keeps every executor's
+samples in one place, next to the jobs and stages they ran alongside, while executor logs are spread
+over the cluster and often go away with their containers.
+
+The driver writes, and flushes, each event of its event log as it goes, so it does not write every
+sample. For each executor, it takes the samples a minute at a time and writes two of them: the one
+with the most untracked memory, and the last, which in the minute an executor goes idle is the one
+after its last native plan finishes. It writes them at the executor's first heartbeat a minute or
+more after the first of them arrived, which comes every `spark.executor.heartbeatInterval`, 10
+seconds by default, busy or not, and when the executor is removed, such as when the cluster manager
+kills it, or the application stops. So each executor adds about two events a minute while it runs
+native plans, however short the interval, and each minute's two reach the event log within about a
+minute of that minute's first sample arriving. The event with the most untracked memory for an
+executor is then its sample with the most, apart from samples it takes while the application stops,
+and samples from the last minute before a driver exits without stopping the application.
+
+Each sample the driver writes is one event, on one line of the event log. It carries the figures of
+the line in bytes, the executor that took the sample, and when it did, in milliseconds since the
+epoch by the executor's clock. The line above looks like this as an event, spread over several lines
+here:
+
+```json
+{
+  "Event": "org.apache.comet.CometExecutorMemoryUsage",
+  "executorId": "3",
+  "time": 1727550000000,
+  "nativeAllocated": 5675212390,
+  "poolsReserved": 4078960640,
+  "pools": 8,
+  "plans": 16,
+  "jvmArrowAllocated": 325478809,
+  "jvmArrowImported": 100873420
+}
+```
+
+`nativeAllocated` is `allocated`, `poolsReserved` is `reserved`, and `jvmArrowAllocated` and
+`jvmArrowImported` are the JVM Arrow figures. With `jq`, for example, this lists the sample with the
+most untracked memory for each executor, in MiB:
+
+```shell
+jq -n -c '[inputs | select(.Event == "org.apache.comet.CometExecutorMemoryUsage")
+    | {executorId, time, untrackedMiB: (([.nativeAllocated - .poolsReserved
+        + ([.jvmArrowAllocated - .jvmArrowImported, 0] | max), 0] | max) / 1048576 | ceil)}]
+  | group_by(.executorId) | map(max_by(.untrackedMiB))[]' <event log files>
+```
+
+From Spark 4.0, event logs are compressed with zstd and split into several files by default, so
+decompress the files first, for example with `zstd -dc`. The Spark history server does not display
+these events, and one without Comet on its classpath logs once that it dropped them. From Spark 4.1,
+`spark.eventLog.excludedPatterns=org.apache.comet.CometExecutorMemoryUsage` leaves them out of the
+event log.
+
 ## Batch Size
 
 Comet processes data in columnar batches. The batch size is controlled by `spark.comet.batchSize` (default
diff --git a/contributor-guide/memory_management.html b/contributor-guide/memory_management.html
index 4238156ea2..2d9ccf9655 100644
--- a/contributor-guide/memory_management.html
+++ b/contributor-guide/memory_management.html
@@ -1138,7 +1138,9 @@ treats it as fatal, so the executor is lost either way and the exit code is what
 them. A failed task with <code class="docutils literal notranslate"><span class="pre">SparkOutOfMemoryError</span></code> and a surviving executor is Spark’s managed
 memory pool, which is the only one of the three that is recoverable at task level.</p></li>
 <li><p>Compare <code class="docutils literal notranslate"><span class="pre">allocated</span></code> against <code class="docutils literal notranslate"><span class="pre">reserved</span></code> in the executor’s <code class="docutils literal notranslate"><span class="pre">Comet</span> <span class="pre">native</span> <span class="pre">memory</span> <span class="pre">usage</span></code> log lines
-leading up to the kill, or <code class="docutils literal notranslate"><span class="pre">native_allocated</span></code> against <code class="docutils literal notranslate"><span class="pre">comet_memory_reserved_total</span></code> in a trace.
+leading up to the kill, or <code class="docutils literal notranslate"><span class="pre">nativeAllocated</span></code> against <code class="docutils literal notranslate"><span class="pre">poolsReserved</span></code> in its
+<code class="docutils literal notranslate"><span class="pre">CometExecutorMemoryUsage</span></code> events when the event log records them, or <code class="docutils literal notranslate"><span class="pre">native_allocated</span></code>
+against <code class="docutils literal notranslate"><span class="pre">comet_memory_reserved_total</span></code> in a trace.
 A large excess points at undeclared native allocations; a small excess points at the budget
 simply being too small, or at the JVM side, which the same lines report as <code class="docutils literal notranslate"><span class="pre">JVM</span> <span class="pre">Arrow</span> <span class="pre">allocated</span></code>.</p></li>
 <li><p>Check <code class="docutils literal notranslate"><span class="pre">spark.comet.batchSize</span></code> against the schema width. Peak memory scales with
diff --git a/contributor-guide/plugin_overview.html b/contributor-guide/plugin_overview.html
index 65c0424cf1..ac691dc8e0 100644
--- a/contributor-guide/plugin_overview.html
+++ b/contributor-guide/plugin_overview.html
@@ -584,6 +584,14 @@ and skips the remaining steps. Otherwise it:</p>
 <p>The plugin does not change any executor memory setting. The <a class="reference internal" href="../user-guide/latest/tuning/memory.html"><span class="std std-doc">Memory Tuning</span></a> guide covers how
 to size them.</p>
 <p>When the driver or an executor stops, the plugin shuts down Comet’s native tokio runtime in that JVM.</p>
+<p>When <code class="docutils literal notranslate"><span class="pre">spark.eventLog.enabled</span></code> is <code class="docutils literal notranslate"><span class="pre">true</span></code>, the executor plugin also carries each sample of the executor’s native memory
+usage log to the driver plugin. For each executor, the driver plugin keeps the sample with the most untracked memory
+and the last sample of every minute, and posts them to Spark’s listener bus as <code class="docutils literal notranslate"><span class="pre">CometExecutorMemoryUsage</span></code> events so
+that the event log records them. A listener on a listener bus queue of its own posts them at the executor’s first
+heartbeat a minute or more after the first of them arrived, which also ends an idle executor’s minute, and posts what
+the driver plugin holds when an executor is removed and when the application ends, since an executor that goes away
+sends nothing more. See
+<a class="reference internal" href="../user-guide/latest/tuning/memory.html#reading-the-memory-usage-log-from-the-event-log"><span class="std std-ref">Reading the Memory Usage Log from the Event Log</span></a>.</p>
 <p><code class="docutils literal notranslate"><span class="pre">CometSparkSessionExtensions</span></code> can also be registered without the plugin, through <code class="docutils literal notranslate"><span class="pre">spark.sql.extensions</span></code> or
 <code class="docutils literal notranslate"><span class="pre">SparkSession.Builder.withExtensions</span></code>. Most of Comet’s test suites and the Spark SQL tests enable Comet this way, so
 none of the driver plugin’s steps run for them.</p>
diff --git a/searchindex.js b/searchindex.js
index 4261839814..788eb39eb5 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[57, "id1"]], "%": [[55, "id1"]], "&": [[45, "id1"]], "*": [[55, "id2"]], "+": [[55, "id3"]], "-": [[55, "id4"]], "/": [[55, "id5"]], "1. Format Your Code": [[42, "format-your-code"]], "1. Install Comet": [[63, "install-comet"], [75, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[42, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[63, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[57, "id1"]], "%": [[55, "id1"]], "&": [[45, "id1"]], "*": [[55, "id2"]], "+": [[55, "id3"]], "-": [[55, "id4"]], "/": [[55, "id5"]], "1. Format Your Code": [[42, "format-your-code"]], "1. Install Comet": [[63, "install-comet"], [75, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[42, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[63, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index a7db417921..af65f83cd5 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -1180,7 +1180,7 @@ shuffle, including applications that rely only on remote reliable storage or dec
 <td><p>107374182400b</p></td>
 </tr>
 <tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.memory.logInterval</span></code></p></td>
-<td><p>How often each executor logs its native memory usage at INFO level while Comet native plans are running: the bytes the native allocator has handed out, the bytes reserved in Comet’s memory pools, and the Arrow memory Comet holds on the JVM side. The difference between the first two is native memory that the pools are not accounting for. The executor logs one line per interval however many tasks are running, and one more after the last plan finishes. It logs a warning when the nati [...]
+<td><p>How often each executor logs its native memory usage at INFO level while Comet native plans are running: the bytes the native allocator has handed out, the bytes reserved in Comet’s memory pools, and the Arrow memory Comet holds on the JVM side. The difference between the first two is native memory that the pools are not accounting for. The executor logs one line per interval however many tasks are running, and one more after the last plan finishes. It logs a warning when the nati [...]
 <td><p>10000ms</p></td>
 </tr>
 <tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.tracing.enabled</span></code></p></td>
diff --git a/user-guide/latest/tuning/memory.html b/user-guide/latest/tuning/memory.html
index 4fef0fa265..c8166b28da 100644
--- a/user-guide/latest/tuning/memory.html
+++ b/user-guide/latest/tuning/memory.html
@@ -899,15 +899,18 @@ operator holds on to one of its batches, such as a sort buffering its input. The
 which allocator a buffer is charged to rather than where it was allocated, so treat the JVM’s
 part as an estimate.</p></li>
 </ul>
+<p>When the application writes an event log and runs the Comet plugin, the event log also records a
+summary of these samples; see <a class="reference internal" href="#reading-the-memory-usage-log-from-the-event-log">Reading the Memory Usage Log from the Event Log</a>.</p>
 <p>Comet’s untracked memory is what Comet holds outside the JVM heap that Spark’s off-heap pool does
 not account for: <code class="docutils literal notranslate"><span class="pre">allocated</span></code>, plus the JVM Arrow figure less the part imported from native, minus
 <code class="docutils literal notranslate"><span class="pre">reserved</span></code>. It is the part of Comet’s footprint that has to fit in <code class="docutils literal notranslate"><span class="pre">spark.executor.memoryOverhead</span></code>,
 alongside the JVM’s own non-heap memory. To size the overhead from it:</p>
 <ol class="arabic simple">
 <li><p>Run a representative workload and find the line with the most untracked memory in each
-executor’s log. Take all the figures from the same line: they are sampled together, and figures
-from different lines describe different moments. Setting <code class="docutils literal notranslate"><span class="pre">spark.comet.memory.logInterval=1s</span></code> for
-this run makes a short-lived peak less likely to fall between samples.</p></li>
+executor’s log, or the event with the most for each executor in the event log. Take all the
+figures from the same line or event: they are sampled together, and figures from different
+samples describe different moments. Setting <code class="docutils literal notranslate"><span class="pre">spark.comet.memory.logInterval=1s</span></code> for this run
+makes a short-lived peak less likely to fall between samples.</p></li>
 <li><p>Start from the overhead the executors had before Comet was enabled, which covers the JVM’s own
 non-heap memory, and add the most untracked memory seen on any executor.</p></li>
 <li><p>Add a margin on top. The log can miss the true peak between samples, and none of the figures
@@ -937,6 +940,56 @@ The executor logs one more line after its last native plan finishes, and an <cod
 <code class="docutils literal notranslate"><span class="pre">JVM</span> <span class="pre">Arrow</span> <span class="pre">allocated</span></code> figure there that grows from one query to the next points the same way.</p>
 <p><code class="docutils literal notranslate"><span class="pre">spark.comet.memory.logInterval</span></code> is read when an executor starts its first Comet native plan, so set
 it when the application is submitted. Set it to <code class="docutils literal notranslate"><span class="pre">0</span></code> to turn the log off.</p>
+<section id="reading-the-memory-usage-log-from-the-event-log">
+<h3>Reading the Memory Usage Log from the Event Log<a class="headerlink" href="#reading-the-memory-usage-log-from-the-event-log" title="Link to this heading">#</a></h3>
+<p>When <code class="docutils literal notranslate"><span class="pre">spark.eventLog.enabled</span></code> is <code class="docutils literal notranslate"><span class="pre">true</span></code> and the application runs the Comet plugin
+(<code class="docutils literal notranslate"><span class="pre">spark.plugins=org.apache.spark.CometPlugin</span></code>), each executor also sends its samples to the driver,
+which writes a summary of them to the application’s event log. The event log keeps every executor’s
+samples in one place, next to the jobs and stages they ran alongside, while executor logs are spread
+over the cluster and often go away with their containers.</p>
+<p>The driver writes, and flushes, each event of its event log as it goes, so it does not write every
+sample. For each executor, it takes the samples a minute at a time and writes two of them: the one
+with the most untracked memory, and the last, which in the minute an executor goes idle is the one
+after its last native plan finishes. It writes them at the executor’s first heartbeat a minute or
+more after the first of them arrived, which comes every <code class="docutils literal notranslate"><span class="pre">spark.executor.heartbeatInterval</span></code>, 10
+seconds by default, busy or not, and when the executor is removed, such as when the cluster manager
+kills it, or the application stops. So each executor adds about two events a minute while it runs
+native plans, however short the interval, and each minute’s two reach the event log within about a
+minute of that minute’s first sample arriving. The event with the most untracked memory for an
+executor is then its sample with the most, apart from samples it takes while the application stops,
+and samples from the last minute before a driver exits without stopping the application.</p>
+<p>Each sample the driver writes is one event, on one line of the event log. It carries the figures of
+the line in bytes, the executor that took the sample, and when it did, in milliseconds since the
+epoch by the executor’s clock. The line above looks like this as an event, spread over several lines
+here:</p>
+<div class="highlight-json notranslate"><div class="highlight"><pre><span></span><span class="p">{</span>
+<span class="w">  </span><span class="nt">"Event"</span><span class="p">:</span><span class="w"> </span><span class="s2">"org.apache.comet.CometExecutorMemoryUsage"</span><span class="p">,</span>
+<span class="w">  </span><span class="nt">"executorId"</span><span class="p">:</span><span class="w"> </span><span class="s2">"3"</span><span class="p">,</span>
+<span class="w">  </span><span class="nt">"time"</span><span class="p">:</span><span class="w"> </span><span class="mi">1727550000000</span><span class="p">,</span>
+<span class="w">  </span><span class="nt">"nativeAllocated"</span><span class="p">:</span><span class="w"> </span><span class="mi">5675212390</span><span class="p">,</span>
+<span class="w">  </span><span class="nt">"poolsReserved"</span><span class="p">:</span><span class="w"> </span><span class="mi">4078960640</span><span class="p">,</span>
+<span class="w">  </span><span class="nt">"pools"</span><span class="p">:</span><span class="w"> </span><span class="mi">8</span><span class="p">,</span>
+<span class="w">  </span><span class="nt">"plans"</span><span class="p">:</span><span class="w"> </span><span class="mi">16</span><span class="p">,</span>
+<span class="w">  </span><span class="nt">"jvmArrowAllocated"</span><span class="p">:</span><span class="w"> </span><span class="mi">325478809</span><span class="p">,</span>
+<span class="w">  </span><span class="nt">"jvmArrowImported"</span><span class="p">:</span><span class="w"> </span><span class="mi">100873420</span>
+<span class="p">}</span>
+</pre></div>
+</div>
+<p><code class="docutils literal notranslate"><span class="pre">nativeAllocated</span></code> is <code class="docutils literal notranslate"><span class="pre">allocated</span></code>, <code class="docutils literal notranslate"><span class="pre">poolsReserved</span></code> is <code class="docutils literal notranslate"><span class="pre">reserved</span></code>, and <code class="docutils literal notranslate"><span class="pre">jvmArrowAllocated</span></code> and
+<code class="docutils literal notranslate"><span class="pre">jvmArrowImported</span></code> are the JVM Arrow figures. With <code class="docutils literal notranslate"><span class="pre">jq</span></code>, for example, this lists the sample with the
+most untracked memory for each executor, in MiB:</p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>jq<span class="w"> </span>-n<span class="w"> </span>-c<span class="w"> </span><span class="s1">'[inputs | select(.Event == "org.apache.comet.CometExecutorMemoryUsage")</span>
+<span class="s1">    | {executorId, time, untrackedMiB: (([.nativeAllocated - .poolsReserved</span>
+<span class="s1">        + ([.jvmArrowAllocated - .jvmArrowImported, 0] | max), 0] | max) / 1048576 | ceil)}]</span>
+<span class="s1">  | group_by(.executorId) | map(max_by(.untrackedMiB))[]'</span><span class="w"> </span>&lt;event<span class="w"> </span>log<span class="w"> </span>files&gt;
+</pre></div>
+</div>
+<p>From Spark 4.0, event logs are compressed with zstd and split into several files by default, so
+decompress the files first, for example with <code class="docutils literal notranslate"><span class="pre">zstd</span> <span class="pre">-dc</span></code>. The Spark history server does not display
+these events, and one without Comet on its classpath logs once that it dropped them. From Spark 4.1,
+<code class="docutils literal notranslate"><span class="pre">spark.eventLog.excludedPatterns=org.apache.comet.CometExecutorMemoryUsage</span></code> leaves them out of the
+event log.</p>
+</section>
 </section>
 <section id="batch-size">
 <h2>Batch Size<a class="headerlink" href="#batch-size" title="Link to this heading">#</a></h2>
@@ -1010,7 +1063,10 @@ multiple of this value.</p>
     <ul class="visible nav section-nav flex-column">
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#configuring-comet-memory">Configuring Comet Memory</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#configuring-executor-memory-overhead">Configuring Executor Memory Overhead</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#sizing-the-overhead-from-the-memory-usage-log">Sizing the Overhead from the Memory Usage Log</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#sizing-the-overhead-from-the-memory-usage-log">Sizing the Overhead from the Memory Usage Log</a><ul class="nav section-nav flex-column">
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#reading-the-memory-usage-log-from-the-event-log">Reading the Memory Usage Log from the Event Log</a></li>
+</ul>
+</li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#batch-size">Batch Size</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#limiting-spill-disk-usage">Limiting Spill Disk Usage</a></li>
 </ul>


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org