You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/01 00:32:11 UTC
(datafusion-comet) branch main updated (f140aedadb -> ec66afe56b)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
from f140aedadb fix: fall back when struct field names collide case-insensitively (#6326)
new 530e08eb03 fix: reject casts involving non-default collated strings (#5302)
new ec66afe56b feat: write the executor memory usage log to the Spark event log (#6401)
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/pr_build_linux.yml | 2 +
.github/workflows/pr_build_macos.yml | 2 +
docs/source/contributor-guide/memory_management.md | 4 +-
docs/source/contributor-guide/plugin_overview.md | 9 +
docs/source/user-guide/latest/tuning/memory.md | 68 ++++-
.../main/scala/org/apache/comet/CometConf.scala | 12 +-
.../scala/org/apache/comet/CometExecIterator.scala | 131 +++++++++-
.../apache/comet/CometExecutorMemoryUsage.scala | 63 +++++
.../comet/codegen/CometBatchKernelCodegen.scala | 15 ++
.../org/apache/comet/expressions/CometCast.scala | 28 +++
.../src/main/scala/org/apache/spark/Plugins.scala | 100 +++++++-
.../spark/CometExecIteratorLifecycleSuite.scala | 70 +++++-
.../scala/org/apache/spark/CometPluginsSuite.scala | 145 ++++++++++-
.../comet/CometCastCollatedStringSuite.scala | 273 +++++++++++++++++++++
14 files changed, 903 insertions(+), 19 deletions(-)
create mode 100644 spark/src/main/scala/org/apache/comet/CometExecutorMemoryUsage.scala
create mode 100644 spark/src/test/spark-4.x/org/apache/comet/CometCastCollatedStringSuite.scala
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 01/02: fix: reject casts involving non-default collated strings (#5302)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 530e08eb03228b2cd60d558f888ee6409f89fc0f
Author: Po-Han (Stan) Shih · 施博瀚 <79...@users.noreply.github.com>
AuthorDate: Wed Sep 30 21:44:12 2026 +0000
fix: reject casts involving non-default collated strings (#5302)
* test: pin down CometCast fallback for non-default collated strings
CometCast.isSupported matches string casts against DataTypes.StringType,
the singleton default-collation instance. A non-default-collation
StringType (e.g. STRING COLLATE UTF8_LCASE) correctly fails that
equality check today and falls back to Spark, but that was implicit
and untested: there was no isStringCollationType guard like the other
string-touching serdes use, and no test pinning the fallback down.
Adds CometCastCollatedStringSuite under spark-4.x (collation is a
Spark 4.0+ feature, shared across every 4.x profile, unlike TimeType
in #4490 which is 4.1-only) asserting isSupported returns Unsupported
for every collated-string pair across LEGACY/TRY/ANSI, plus two
Compatible() sanity baselines (same-collation identity cast, and
default-collation identity cast) documenting the boundary this issue
is not about: an identity cast is a byte-for-byte no-op regardless of
collation, so Compatible() there is correct, not a gap.
Closes #4489
* style: apply spotless formatting to CometCastCollatedStringSuite
Rewraps the Scaladoc comment block to match what 'mvn spotless:apply'
(scalafmt) produces. Verified via a real mvn test -Pspark-4.1 run in
WSL (spotless:check now passes; 7/7 tests still pass).
* fix: reject casts involving non-default collated strings
CometCast.isSupported only failed to match a collated StringType
because DataTypes.StringType is the default-collation singleton and
Scala pattern equality compares the whole instance. The
fromType == toType shortcut let identity casts through regardless,
including nested ones such as ARRAY<STRING COLLATE UTF8_LCASE>, and
serializeDataType maps every StringType to one proto type id, so the
collation was dropped from the plan with no warning.
Reject collated source and target types up front using the existing
CometTypeShim.hasNonDefaultStringCollation, the same helper the array,
collection, datetime, map, predicate, and string serdes already use.
It walks nested element, key, value, and field types, and is stubbed
to false on Spark 3.x where collation does not exist.
Rework CometCastCollatedStringSuite accordingly. Unsupported means
there is no native path rather than a fallback to Spark, since
CometCast mixes in CodegenDispatchFallback, so the test names and the
Scaladoc now say that. Adds nested coverage for arrays, structs, and
maps, over-block checks for default-collation casts, and end-to-end
coverage of both settings of spark.comet.exec.scalaUDF.codegen.enabled.
Closes #4489
* test: cover the cast pairs the collation guard newly blocks
Working out which pairs actually changed answer turned up three that
were Compatible on main and are not identity casts, so the guard's
over-block surface was wider than the first revision of this suite
described.
A struct whose collated field is unchanged while a sibling field is
cast answered Compatible, because the field zip answered per field and
the collated field matched the fromType == toType shortcut. A map with
an unchanged collated key and a cast value did the same through the
key. ArrayType(NullType) -> ArrayType(lcase) answered Compatible
through the elementType == NullType branch, which runs ahead of
everything else.
The struct case also gives the suite its first end-to-end test that
fails without the guard. The sibling field changes type, so the cast
survives SimplifyCasts and the collated field rides along inside it.
A scalar identity cast cannot be reached from SQL at all, since
SimplifyCasts drops a cast whose child already has the target type and
the query arrives at the planner as a bare Collate. There is a comment
in the suite recording that, in the style of the unreachable join
tests in CometCollationSuite.
* fix: give the collation cast rejection its own fallback reason
The guard returned the generic `Cast from $fromType to $toType is not
supported` template. For an identity cast both sides print the same, so
EXPLAIN read like a nonsensical refusal of a string-to-string cast with
no hint that collation was the cause. Name the reason on `CometCast` and
have the suite read it from production so the two cannot drift.
Also document why `CometBatchKernelCodegen.isSupportedDataType` admits a
collated `StringType`. The dispatcher's declared return type does go
through `serializeDataType`, which flattens collation, but every
collation-sensitive decision Comet makes reads the Catalyst `DataType`
and blocks the operator before a native kernel sees the column.
* test: describe the struct end-to-end case as a fallback moved into the dispatcher
On main the struct cast query already falls back to Spark, because
Collate has no serde. The guard moves it into the codegen dispatcher
when that is on, and puts the collation reason on the plan when it is
off. Also fix the end-to-end test count, and reword three matrix
comments (identity cast, unchanged field beside a cast sibling, array
of nulls) that implied a collated type reached the native side. They
now describe what isSupported used to answer. The array-of-nulls
comment also no longer says its branch runs ahead of every other one,
since (NullType, _) and the identity shortcut come first.
Co-Authored-By: Claude Opus 5.5 <no...@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <no...@anthropic.com>
---
.github/workflows/pr_build_linux.yml | 1 +
.github/workflows/pr_build_macos.yml | 1 +
.../comet/codegen/CometBatchKernelCodegen.scala | 15 ++
.../org/apache/comet/expressions/CometCast.scala | 28 +++
.../comet/CometCastCollatedStringSuite.scala | 273 +++++++++++++++++++++
5 files changed, 318 insertions(+)
diff --git a/.github/workflows/pr_build_linux.yml b/.github/workflows/pr_build_linux.yml
index ee8a947aa4..7341c32308 100644
--- a/.github/workflows/pr_build_linux.yml
+++ b/.github/workflows/pr_build_linux.yml
@@ -630,6 +630,7 @@ jobs:
org.apache.comet.CometTemporalExpressionSuite
org.apache.comet.CometArrayExpressionSuite
org.apache.comet.CometNativeCastSuite
+ org.apache.comet.CometCastCollatedStringSuite
org.apache.comet.CometDateTimeUtilsSuite
org.apache.comet.CometMathExpressionSuite
org.apache.comet.CometStringExpressionSuite
diff --git a/.github/workflows/pr_build_macos.yml b/.github/workflows/pr_build_macos.yml
index 371e4f6531..6c7ba5b367 100644
--- a/.github/workflows/pr_build_macos.yml
+++ b/.github/workflows/pr_build_macos.yml
@@ -287,6 +287,7 @@ jobs:
org.apache.comet.CometTemporalExpressionSuite
org.apache.comet.CometArrayExpressionSuite
org.apache.comet.CometNativeCastSuite
+ org.apache.comet.CometCastCollatedStringSuite
org.apache.comet.CometDateTimeUtilsSuite
org.apache.comet.CometMathExpressionSuite
org.apache.comet.CometStringExpressionSuite
diff --git a/spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegen.scala b/spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegen.scala
index 83fbca6b63..eadda24da7 100644
--- a/spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegen.scala
+++ b/spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegen.scala
@@ -87,6 +87,21 @@ object CometBatchKernelCodegen extends Logging with CometExprTraitShim with Come
* single child and the generated writer NPEs on the missing ordinal-1 vector.
* `CometCreateNamedStruct` declines them on the native path for the same reason, but a struct
* nested inside a dispatcher-built value (a `CreateMap` value) never reaches that check.
+ *
+ * The `StringType` case admits non-default collations (Spark 4+), on purpose. Collation is
+ * carried by the expression, not by the value: the kernel runs Spark's own `doGenCode` against
+ * the bound tree, whose `collationId` survives closure serialization, and Arrow is only the
+ * byte store for the `UTF8String`s that code produces. So a dispatched expression over collated
+ * input answers exactly as Spark does.
+ *
+ * The proto type is a separate matter. `CometScalaUDF.emitJvmCodegenDispatch` declares the
+ * return type through `QueryPlanSerde.serializeDataType`, which flattens every `StringType` to
+ * one proto id, so the native plan describes a dispatched collated output as a plain string.
+ * Nothing on this route reads that back. The values are bytes and the kernel is what produced
+ * them. Whether a downstream operator may then treat the column collation-blind is decided per
+ * operator against the Catalyst `DataType`, which keeps its collation, and does not depend on
+ * what this predicate admits. Rejecting collated strings here would force a full Spark fallback
+ * for a route that is already correct.
*/
def isSupportedDataType(dt: DataType): Boolean = dt match {
case BooleanType | ByteType | ShortType | IntegerType | LongType => true
diff --git a/spark/src/main/scala/org/apache/comet/expressions/CometCast.scala b/spark/src/main/scala/org/apache/comet/expressions/CometCast.scala
index d5502b823e..e14fb086b1 100644
--- a/spark/src/main/scala/org/apache/comet/expressions/CometCast.scala
+++ b/spark/src/main/scala/org/apache/comet/expressions/CometCast.scala
@@ -50,6 +50,16 @@ object CometCast
private[comet] val legacyCastComplexTypesToStringReason: String =
"spark.sql.legacy.castComplexTypesToString.enabled=true is not supported"
+ // The generic `Cast from $fromType to $toType is not supported` template is unhelpful for a
+ // collation rejection. An identity cast prints both sides identically, so it reads like a
+ // nonsensical refusal of a string-to-string cast unless the reader already knows collation is
+ // the cause. Named here, and shared with `CometCastCollatedStringSuite`, so the asserted reason
+ // cannot drift from production. Follows the phrasing the other collation reasons use, e.g.
+ // `CometReverse` in `collectionOperations.scala` and `ComparisonUtils` in `predicates.scala`.
+ private[comet] val nonDefaultCollationReason: String =
+ "Cast involving a non-default string collation is not supported " +
+ "(https://github.com/apache/datafusion-comet/issues/4489)"
+
private def legacyCastComplexTypesToString: Boolean =
SQLConf.get
.getConfString("spark.sql.legacy.castComplexTypesToString.enabled", "false")
@@ -191,6 +201,24 @@ object CometCast
return unsupported(fromType, toType)
}
+ // Spark 4.0's collation metadata rides on `StringType`, but `serializeDataType` maps every
+ // `StringType` to the same proto id, so a non-default collation is dropped on the way into
+ // the native plan with no warning. Reject the cast outright rather than relying on the
+ // pattern matching below, which only misses collated types because `DataTypes.StringType` is
+ // the default-collation singleton and Scala pattern equality happens not to match. This runs
+ // above the `fromType == toType` shortcut so that an identity cast on a collated type is
+ // checked too, and `hasNonDefaultStringCollation` walks nested element, key, value, and field
+ // types. The version-shimmed helper returns false on Spark 3.x, where collation does not
+ // exist. See https://github.com/apache/datafusion-comet/issues/4489.
+ //
+ // `Unsupported` here means there is no native path, not that the plan falls back to Spark.
+ // `CodegenDispatchFallback` offers the cast to the JVM codegen dispatcher first, and that
+ // route is result-correct: see the note on `isSupportedDataType` in
+ // `CometBatchKernelCodegen` for why a collated string is safe to admit there.
+ if (hasNonDefaultStringCollation(fromType) || hasNonDefaultStringCollation(toType)) {
+ return Unsupported(Some(nonDefaultCollationReason))
+ }
+
if (fromType == toType) {
return Compatible()
}
diff --git a/spark/src/test/spark-4.x/org/apache/comet/CometCastCollatedStringSuite.scala b/spark/src/test/spark-4.x/org/apache/comet/CometCastCollatedStringSuite.scala
new file mode 100644
index 0000000000..f547836a7b
--- /dev/null
+++ b/spark/src/test/spark-4.x/org/apache/comet/CometCastCollatedStringSuite.scala
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet
+
+import org.apache.spark.sql.CometTestBase
+import org.apache.spark.sql.types.{ArrayType, DataType, DataTypes, IntegerType, MapType, StringType, StructField, StructType}
+
+import org.apache.comet.expressions.{CometCast, CometEvalMode}
+import org.apache.comet.serde.{Compatible, Unsupported}
+
+/**
+ * https://github.com/apache/datafusion-comet/issues/4489
+ *
+ * Spark 4.0 carries collation metadata on `StringType`, but `serializeDataType` maps every
+ * `StringType` to one proto type id, so the collation is dropped on the way into the native plan
+ * with no warning. Nothing used to stop that from happening. `CometCast.isSupported` matched
+ * string casts through `case (DataTypes.StringType, _)` and `case (_, DataTypes.StringType)`, and
+ * those only failed to match a collated `StringType` because `DataTypes.StringType` is the
+ * default-collation singleton and Scala pattern equality compares the whole instance. The right
+ * answer fell out of an accident of pattern matching. The `fromType == toType` shortcut let
+ * identity casts on collated types through regardless, including nested ones such as
+ * `ARRAY<STRING COLLATE UTF8_LCASE>`, which were byte-safe only because the cast was a no-op.
+ *
+ * `isSupported` now rejects any source or target type carrying a non-default collation before
+ * either of those paths runs. This suite pins the resulting matrix down so the behaviour no
+ * longer depends on pattern-match semantics staying the way they are.
+ *
+ * A note on what `Unsupported` means here. It does not mean the query falls back to Spark.
+ * `CometCast` mixes in `CodegenDispatchFallback`, so `QueryPlanSerde.exprToProtoInternal` offers
+ * the expression to the JVM codegen dispatcher first, which runs Spark's own `doGenCode` inside
+ * the Comet pipeline. `spark.comet.exec.scalaUDF.codegen.enabled` defaults to true and
+ * `CometBatchKernelCodegen.isSupportedDataType` admits every `StringType` regardless of
+ * collation, so under default config a collated cast usually stays inside Comet. That route is
+ * result-correct, because the kernel evaluates Spark's own generated code and the `collationId`
+ * rides along on the expression. `Unsupported` means there is no native path. The end-to-end
+ * tests at the bottom run both settings of that config on both the scalar and the struct case.
+ *
+ * These are Scala tests rather than `CometSqlFileTestSuite` fixtures because most of the file
+ * asserts on `CometCast.isSupported` over type pairs no SQL can construct, such as
+ * `ArrayType(NullType) -> ArrayType(STRING COLLATE UTF8_LCASE)`. The end-to-end tests stay here
+ * with them because `--Config` is file scoped, so the query that has to assert a fallback reason
+ * with the dispatcher off and the query that has to assert native execution with it on could not
+ * share a fixture file.
+ *
+ * This lives under `spark-4.x`, shared by every 4.x profile, rather than `spark-4.1+`, because
+ * collation is a Spark 4.0 feature and `StringType(collationName)` already resolves there.
+ */
+class CometCastCollatedStringSuite extends CometTestBase {
+
+ private val lcase = StringType("UTF8_LCASE")
+ private val unicode = StringType("UNICODE")
+
+ private val evalModes = Seq(CometEvalMode.LEGACY, CometEvalMode.TRY, CometEvalMode.ANSI)
+
+ private def structWith(dt: DataType): StructType = StructType(Seq(StructField("s", dt)))
+
+ /** Asserts that Comet reports no native path for this cast under every eval mode. */
+ private def assertNoNativePath(fromType: DataType, toType: DataType): Unit = {
+ evalModes.foreach { evalMode =>
+ CometCast.isSupported(fromType, toType, None, evalMode) match {
+ case _: Unsupported => // expected
+ case other =>
+ fail(s"expected Unsupported for $fromType -> $toType under $evalMode, got $other")
+ }
+ }
+ }
+
+ /** Asserts that the collation guard leaves an uncollated cast alone. */
+ private def assertCompatible(fromType: DataType, toType: DataType): Unit = {
+ evalModes.foreach { evalMode =>
+ CometCast.isSupported(fromType, toType, None, evalMode) match {
+ case _: Compatible => // expected
+ case other =>
+ fail(s"expected Compatible for $fromType -> $toType under $evalMode, got $other")
+ }
+ }
+ }
+
+ // ---- scalar collated strings ----------------------------------------------------
+
+ test("cast collated string to IntegerType has no native path") {
+ assertNoNativePath(lcase, IntegerType)
+ }
+
+ test("cast IntegerType to collated string has no native path") {
+ // This pair leaves through `canCastFromInt`'s catch-all rather than `canCastToString`,
+ // because `case (_, DataTypes.StringType)` does not match a collated target. The guard now
+ // answers ahead of both.
+ assertNoNativePath(IntegerType, lcase)
+ }
+
+ test("cast collated string to default-collation StringType has no native path") {
+ assertNoNativePath(lcase, DataTypes.StringType)
+ }
+
+ test("cast default-collation StringType to collated string has no native path") {
+ assertNoNativePath(DataTypes.StringType, lcase)
+ }
+
+ test("cast between two different collations has no native path") {
+ assertNoNativePath(lcase, unicode)
+ }
+
+ test("cast collated string to the same collation has no native path") {
+ // The `fromType == toType` shortcut answered `Compatible()` here before the guard existed.
+ // The result would be right, since the cast is a byte-level no-op, but `isSupported` was
+ // clearing a collated type for a proto that cannot carry the collation, and nothing recorded
+ // that. This is the implicit behaviour #4489 names, so the guard sits above the shortcut.
+ assertNoNativePath(lcase, lcase)
+ }
+
+ // ---- nested collated strings ----------------------------------------------------
+
+ test("cast array of collated strings to another collation has no native path") {
+ assertNoNativePath(ArrayType(lcase), ArrayType(unicode))
+ }
+
+ test("cast array of collated strings to the same collation has no native path") {
+ // Same identity shortcut as the scalar case, one level down.
+ assertNoNativePath(ArrayType(lcase), ArrayType(lcase))
+ }
+
+ test("cast array of collated strings to StringType has no native path") {
+ // `case (dt: ArrayType, DataTypes.StringType)` recurses on the element type, so the reason
+ // used to describe the element rather than the array. The guard answers for the whole type.
+ assertNoNativePath(ArrayType(lcase), DataTypes.StringType)
+ }
+
+ test("cast struct with a collated field has no native path") {
+ assertNoNativePath(structWith(lcase), structWith(unicode))
+ assertNoNativePath(structWith(lcase), structWith(lcase))
+ }
+
+ test("cast map with a collated key has no native path") {
+ assertNoNativePath(MapType(lcase, IntegerType), MapType(unicode, IntegerType))
+ assertNoNativePath(MapType(lcase, IntegerType), MapType(lcase, IntegerType))
+ }
+
+ test("cast map with a collated value has no native path") {
+ assertNoNativePath(MapType(IntegerType, lcase), MapType(IntegerType, unicode))
+ assertNoNativePath(MapType(IntegerType, lcase), MapType(IntegerType, lcase))
+ }
+
+ test("cast struct whose collated field is unchanged while a sibling field is cast") {
+ // The field zip used to answer per field, so the collated field hit the identity shortcut
+ // and reported Compatible while the sibling carried the cast. That cleared a collated field
+ // for the native plan on another field's back. The guard answers for the whole struct.
+ val from = StructType(Seq(StructField("a", IntegerType), StructField("s", lcase)))
+ val to = StructType(Seq(StructField("a", DataTypes.StringType), StructField("s", lcase)))
+ assertNoNativePath(from, to)
+ }
+
+ test("cast map whose collated key is unchanged while the value type is cast") {
+ assertNoNativePath(MapType(lcase, IntegerType), MapType(lcase, DataTypes.LongType))
+ }
+
+ test("cast array of nulls to array of collated strings has no native path") {
+ // `case (dt: ArrayType, _: ArrayType) if dt.elementType == NullType` returns Compatible
+ // without looking at the target element type, so this was one more pair where
+ // `isSupported` cleared a collated target type.
+ assertNoNativePath(ArrayType(DataTypes.NullType), ArrayType(lcase))
+ }
+
+ // ---- the guard must not over-block ----------------------------------------------
+
+ test("default-collation string casts are untouched by the collation guard") {
+ assertCompatible(DataTypes.StringType, DataTypes.StringType)
+ assertCompatible(DataTypes.StringType, IntegerType)
+ assertCompatible(IntegerType, DataTypes.StringType)
+ }
+
+ test("nested default-collation string casts are untouched by the collation guard") {
+ assertCompatible(ArrayType(DataTypes.StringType), ArrayType(DataTypes.StringType))
+ assertCompatible(structWith(DataTypes.StringType), structWith(DataTypes.StringType))
+ assertCompatible(
+ MapType(DataTypes.StringType, IntegerType),
+ MapType(DataTypes.StringType, IntegerType))
+ }
+
+ // ---- end to end -----------------------------------------------------------------
+ //
+ // The matrix above only exercises `isSupported`. These four run a query and pin down what the
+ // planner actually does with the answer, which is what #4489 asked for. A plain-string Parquet
+ // column with `COLLATE` applied on top reaches the cast as a collated child, the same shape
+ // the datetime tests in `CometCollationSuite` rely on. The cast child has to be a column
+ // rather than a literal, since `getSupportLevel` folds literal children before `isSupported`
+ // is consulted.
+
+ private def withCollatedTable(f: => Unit): Unit =
+ withParquetTable(Seq(("123", 1), ("456", 2)), "collated_cast_tbl")(f)
+
+ // Read from production rather than retyped, so the two cannot drift apart. The guard is the
+ // only thing that produces this reason: without it `StringType(UTF8_LCASE) -> IntegerType`
+ // exits through `isSupported`'s `case _` catch-all and reports the generic
+ // `Cast from ... to ... is not supported` template instead.
+ private val collationReason = CometCast.nonDefaultCollationReason
+
+ // A scalar identity pair such as `lcase -> lcase` cannot be reached from SQL. Spark's
+ // `SimplifyCasts` drops a cast whose child already carries the target type, so
+ // `CAST(_1 COLLATE utf8_lcase AS STRING COLLATE UTF8_LCASE)` arrives at the planner as a bare
+ // `Collate` and the only fallback reason on the plan is "collate is not supported", which comes
+ // from a different serde. Those pairs stay pinned at the `isSupported` level above, in the same
+ // spirit as the join tests in `CometCollationSuite` that no query can reach. A struct is the
+ // way in: when a sibling field changes type the cast survives, and the collated field rides
+ // along inside it. That case is covered end to end below.
+
+ test("cast from a collated string falls back to Spark when codegen dispatch is off") {
+ withCollatedTable {
+ withSQLConf(CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key -> "false") {
+ checkSparkAnswerAndFallbackReason(
+ "SELECT CAST(_1 COLLATE utf8_lcase AS INT) FROM collated_cast_tbl",
+ collationReason)
+ }
+ }
+ }
+
+ test("cast from a collated string routes through the codegen dispatcher when it is on") {
+ withCollatedTable {
+ withSQLConf(CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key -> "true") {
+ checkSparkAnswerAndOperator(
+ "SELECT CAST(_1 COLLATE utf8_lcase AS INT) FROM collated_cast_tbl")
+ }
+ }
+ }
+
+ test("cast of a struct carrying a collated field has no native path end to end") {
+ // A pair the guard changes the outcome for, not just the reason string. The sibling field
+ // changes type so the cast survives `SimplifyCasts`. Without the guard the field zip
+ // answered `Compatible`, since the collated field matched the identity shortcut, but the
+ // query still fell back to Spark: `Collate` has no serde, so serializing the struct stopped
+ // at the collated field with "collate is not supported". With the guard the cast itself
+ // is `Unsupported`: with the dispatcher off the plan carries the collation reason, and with
+ // it on (next test) the query moves out of that Spark fallback and into the dispatcher.
+ withCollatedTable {
+ withSQLConf(CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key -> "false") {
+ checkSparkAnswerAndFallbackReason(
+ "SELECT CAST(struct(_2 AS a, _1 COLLATE utf8_lcase AS s) AS " +
+ "STRUCT<a: STRING, s: STRING COLLATE UTF8_LCASE>) FROM collated_cast_tbl",
+ collationReason)
+ }
+ }
+ }
+
+ test("cast of a struct carrying a collated field routes through the codegen dispatcher") {
+ // The struct is the one case the guard redirects rather than merely relabels, so it needs the
+ // dispatcher-on half too. `isSupportedDataType` recurses into the fields and accepts them, so
+ // the cast stays in the Comet pipeline and the kernel runs Spark's own `Cast.doGenCode`.
+ withCollatedTable {
+ withSQLConf(CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key -> "true") {
+ checkSparkAnswerAndOperator(
+ "SELECT CAST(struct(_2 AS a, _1 COLLATE utf8_lcase AS s) AS " +
+ "STRUCT<a: STRING, s: STRING COLLATE UTF8_LCASE>) FROM collated_cast_tbl")
+ }
+ }
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 02/02: feat: write the executor memory usage log to the Spark event log (#6401)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit ec66afe56b1b9eb5ecc695169feb26495917eedd
Author: Oleks V <co...@users.noreply.github.com>
AuthorDate: Wed Sep 30 22:00:57 2026 +0000
feat: write the executor memory usage log to the Spark event log (#6401)
* feat: write the executor memory usage log to the Spark event log
When spark.eventLog.enabled is true and the application runs the Comet
plugin, each executor sends its periodic native memory usage sample to
the driver plugin, which posts it to the listener bus as a
CometExecutorMemoryUsage event. The event log then keeps every
executor's samples in one place, and the executor logs the line at
DEBUG only.
Without the plugin, or after a failed send, the line stays in the
executor's log at INFO as before. The warning about exceeding the
executor's container always goes to the executor's log.
* Summarize memory usage samples on the driver, keep the INFO log
Addresses the review of the event log path.
The executor logs every memory usage sample at INFO again, so the
event log only adds to the executor's log. It still sends each sample
to the driver.
The driver keeps, for each executor, the sample with the most
untracked memory and the last sample of each minute, and writes only
those. EventLoggingListener flushes every event of this kind, so the
event log now gets about two events per busy executor per minute
however short the interval, rather than one per interval.
The driver holds the samples rather than the executor, because
stopExecutors() sends StopExecutor one-way and the driver stops its
listener bus shortly after, so an executor cannot send what it holds
when the application stops. A listener on a listener bus queue of its
own writes what the driver holds when an executor is removed and when
the application ends. It has its own queue because on Spark 3.4 and
3.5 the bus stops before the plugins shut down, and a slow listener on
the shared queue could otherwise hold the application's end back until
the bus has stopped. From Spark 4.0 the plugin's shutdown writes what
is left as well.
The executor plugin reads spark.eventLog.enabled through Spark's typed
entry, as the driver does, so a value with surrounding whitespace no
longer stops the memory usage log. The tuning guide's jq recipe clamps
untracked memory at zero, as the code does.
* End memory usage summaries at executor heartbeats
Addresses the review of the idle executor case.
An executor that went idle kept its last minute of memory usage samples
on the driver until it ran native plans again, was removed, or the
application stopped, which on a long-lived application can be hours.
The driver plugin's listener now ends an executor's summary at the
executor's first heartbeat a minute after the driver received the first
sample of it. Executors send a heartbeat every
spark.executor.heartbeatInterval whether or not they are busy, and the
event log does not record them. That is now the only place a minute
ends, by the driver's monotonic clock, so receive only adds samples and
the executor's clock no longer matters.
The driver plugin takes a Spark Clock, as HeartbeatReceiver does, so
that a test can move a ManualClock on a minute.
* Say which samples reach the event log within a minute
The tuning guide said the event log has every sample within about a
minute of its arrival, but it records only each minute's peak and last.
It now says those two reach the event log within about a minute of that
minute's first sample arriving.
* Assert memory usage samples in the event log against a marker
The event log tests asserted that the driver plugin's listener records a
sample after the event that makes it record it. The listener bus hands an
event to one queue at a time, and the Comet queue comes before the event
log's, so the listener's samples can reach the event log ahead of that
event. Spark 4.2's exec job failed that way.
Each test now posts a marker once every queue has handled the event, and
asserts that the event log records the sample exactly once, ahead of the
marker. The context posts its own application end only after the marker.
---
.github/workflows/pr_build_linux.yml | 1 +
.github/workflows/pr_build_macos.yml | 1 +
docs/source/contributor-guide/memory_management.md | 4 +-
docs/source/contributor-guide/plugin_overview.md | 9 ++
docs/source/user-guide/latest/tuning/memory.md | 68 +++++++++-
.../main/scala/org/apache/comet/CometConf.scala | 12 +-
.../scala/org/apache/comet/CometExecIterator.scala | 131 ++++++++++++++++++-
.../apache/comet/CometExecutorMemoryUsage.scala | 63 +++++++++
.../src/main/scala/org/apache/spark/Plugins.scala | 100 +++++++++++++-
.../spark/CometExecIteratorLifecycleSuite.scala | 70 +++++++++-
.../scala/org/apache/spark/CometPluginsSuite.scala | 145 ++++++++++++++++++++-
11 files changed, 585 insertions(+), 19 deletions(-)
diff --git a/.github/workflows/pr_build_linux.yml b/.github/workflows/pr_build_linux.yml
index 7341c32308..84800d9064 100644
--- a/.github/workflows/pr_build_linux.yml
+++ b/.github/workflows/pr_build_linux.yml
@@ -583,6 +583,7 @@ jobs:
org.apache.spark.CometPluginsUnifiedModeSuite
org.apache.spark.CometPluginsExtensionOnlySuite
org.apache.spark.CometPluginsSparkShuffleManagerSuite
+ org.apache.spark.CometPluginsEventLogSuite
org.apache.comet.rules.CometScanRuleSuite
org.apache.comet.rules.CometScanContribSuite
org.apache.comet.rules.CometScanSchemeFallbackSuite
diff --git a/.github/workflows/pr_build_macos.yml b/.github/workflows/pr_build_macos.yml
index 6c7ba5b367..3660aa8884 100644
--- a/.github/workflows/pr_build_macos.yml
+++ b/.github/workflows/pr_build_macos.yml
@@ -240,6 +240,7 @@ jobs:
org.apache.spark.CometPluginsUnifiedModeSuite
org.apache.spark.CometPluginsExtensionOnlySuite
org.apache.spark.CometPluginsSparkShuffleManagerSuite
+ org.apache.spark.CometPluginsEventLogSuite
org.apache.comet.rules.CometScanRuleSuite
org.apache.comet.rules.CometScanContribSuite
org.apache.comet.rules.CometScanSchemeFallbackSuite
diff --git a/docs/source/contributor-guide/memory_management.md b/docs/source/contributor-guide/memory_management.md
index a699768379..d899d02029 100644
--- a/docs/source/contributor-guide/memory_management.md
+++ b/docs/source/contributor-guide/memory_management.md
@@ -572,7 +572,9 @@ A checklist for triaging an executor OOM kill:
them. A failed task with `SparkOutOfMemoryError` and a surviving executor is Spark's managed
memory pool, which is the only one of the three that is recoverable at task level.
2. Compare `allocated` against `reserved` in the executor's `Comet native memory usage` log lines
- leading up to the kill, or `native_allocated` against `comet_memory_reserved_total` in a trace.
+ leading up to the kill, or `nativeAllocated` against `poolsReserved` in its
+ `CometExecutorMemoryUsage` events when the event log records them, or `native_allocated`
+ against `comet_memory_reserved_total` in a trace.
A large excess points at undeclared native allocations; a small excess points at the budget
simply being too small, or at the JVM side, which the same lines report as `JVM Arrow allocated`.
3. Check `spark.comet.batchSize` against the schema width. Peak memory scales with
diff --git a/docs/source/contributor-guide/plugin_overview.md b/docs/source/contributor-guide/plugin_overview.md
index a97cc15f41..89586bf7b6 100644
--- a/docs/source/contributor-guide/plugin_overview.md
+++ b/docs/source/contributor-guide/plugin_overview.md
@@ -57,6 +57,15 @@ to size them.
When the driver or an executor stops, the plugin shuts down Comet's native tokio runtime in that JVM.
+When `spark.eventLog.enabled` is `true`, the executor plugin also carries each sample of the executor's native memory
+usage log to the driver plugin. For each executor, the driver plugin keeps the sample with the most untracked memory
+and the last sample of every minute, and posts them to Spark's listener bus as `CometExecutorMemoryUsage` events so
+that the event log records them. A listener on a listener bus queue of its own posts them at the executor's first
+heartbeat a minute or more after the first of them arrived, which also ends an idle executor's minute, and posts what
+the driver plugin holds when an executor is removed and when the application ends, since an executor that goes away
+sends nothing more. See
+[Reading the Memory Usage Log from the Event Log](../user-guide/latest/tuning/memory.md#reading-the-memory-usage-log-from-the-event-log).
+
`CometSparkSessionExtensions` can also be registered without the plugin, through `spark.sql.extensions` or
`SparkSession.Builder.withExtensions`. Most of Comet's test suites and the Spark SQL tests enable Comet this way, so
none of the driver plugin's steps run for them.
diff --git a/docs/source/user-guide/latest/tuning/memory.md b/docs/source/user-guide/latest/tuning/memory.md
index 9b38348c42..1f6ea98a2b 100644
--- a/docs/source/user-guide/latest/tuning/memory.md
+++ b/docs/source/user-guide/latest/tuning/memory.md
@@ -157,15 +157,19 @@ Comet native memory usage: allocated 5412.3 MiB, reserved 3890.0 MiB (16 native
which allocator a buffer is charged to rather than where it was allocated, so treat the JVM's
part as an estimate.
+When the application writes an event log and runs the Comet plugin, the event log also records a
+summary of these samples; see [Reading the Memory Usage Log from the Event Log].
+
Comet's untracked memory is what Comet holds outside the JVM heap that Spark's off-heap pool does
not account for: `allocated`, plus the JVM Arrow figure less the part imported from native, minus
`reserved`. It is the part of Comet's footprint that has to fit in `spark.executor.memoryOverhead`,
alongside the JVM's own non-heap memory. To size the overhead from it:
1. Run a representative workload and find the line with the most untracked memory in each
- executor's log. Take all the figures from the same line: they are sampled together, and figures
- from different lines describe different moments. Setting `spark.comet.memory.logInterval=1s` for
- this run makes a short-lived peak less likely to fall between samples.
+ executor's log, or the event with the most for each executor in the event log. Take all the
+ figures from the same line or event: they are sampled together, and figures from different
+ samples describe different moments. Setting `spark.comet.memory.logInterval=1s` for this run
+ makes a short-lived peak less likely to fall between samples.
2. Start from the overhead the executors had before Comet was enabled, which covers the JVM's own
non-heap memory, and add the most untracked memory seen on any executor.
3. Add a margin on top. The log can miss the true peak between samples, and none of the figures
@@ -199,6 +203,64 @@ The executor logs one more line after its last native plan finishes, and an `all
`spark.comet.memory.logInterval` is read when an executor starts its first Comet native plan, so set
it when the application is submitted. Set it to `0` to turn the log off.
+[Reading the Memory Usage Log from the Event Log]: #reading-the-memory-usage-log-from-the-event-log
+
+### Reading the Memory Usage Log from the Event Log
+
+When `spark.eventLog.enabled` is `true` and the application runs the Comet plugin
+(`spark.plugins=org.apache.spark.CometPlugin`), each executor also sends its samples to the driver,
+which writes a summary of them to the application's event log. The event log keeps every executor's
+samples in one place, next to the jobs and stages they ran alongside, while executor logs are spread
+over the cluster and often go away with their containers.
+
+The driver writes, and flushes, each event of its event log as it goes, so it does not write every
+sample. For each executor, it takes the samples a minute at a time and writes two of them: the one
+with the most untracked memory, and the last, which in the minute an executor goes idle is the one
+after its last native plan finishes. It writes them at the executor's first heartbeat a minute or
+more after the first of them arrived, which comes every `spark.executor.heartbeatInterval`, 10
+seconds by default, busy or not, and when the executor is removed, such as when the cluster manager
+kills it, or the application stops. So each executor adds about two events a minute while it runs
+native plans, however short the interval, and each minute's two reach the event log within about a
+minute of that minute's first sample arriving. The event with the most untracked memory for an
+executor is then its sample with the most, apart from samples it takes while the application stops,
+and samples from the last minute before a driver exits without stopping the application.
+
+Each sample the driver writes is one event, on one line of the event log. It carries the figures of
+the line in bytes, the executor that took the sample, and when it did, in milliseconds since the
+epoch by the executor's clock. The line above looks like this as an event, spread over several lines
+here:
+
+```json
+{
+ "Event": "org.apache.comet.CometExecutorMemoryUsage",
+ "executorId": "3",
+ "time": 1727550000000,
+ "nativeAllocated": 5675212390,
+ "poolsReserved": 4078960640,
+ "pools": 8,
+ "plans": 16,
+ "jvmArrowAllocated": 325478809,
+ "jvmArrowImported": 100873420
+}
+```
+
+`nativeAllocated` is `allocated`, `poolsReserved` is `reserved`, and `jvmArrowAllocated` and
+`jvmArrowImported` are the JVM Arrow figures. With `jq`, for example, this lists the sample with the
+most untracked memory for each executor, in MiB:
+
+```shell
+jq -n -c '[inputs | select(.Event == "org.apache.comet.CometExecutorMemoryUsage")
+ | {executorId, time, untrackedMiB: (([.nativeAllocated - .poolsReserved
+ + ([.jvmArrowAllocated - .jvmArrowImported, 0] | max), 0] | max) / 1048576 | ceil)}]
+ | group_by(.executorId) | map(max_by(.untrackedMiB))[]' <event log files>
+```
+
+From Spark 4.0, event logs are compressed with zstd and split into several files by default, so
+decompress the files first, for example with `zstd -dc`. The Spark history server does not display
+these events, and one without Comet on its classpath logs once that it dropped them. From Spark 4.1,
+`spark.eventLog.excludedPatterns=org.apache.comet.CometExecutorMemoryUsage` leaves them out of the
+event log.
+
## Batch Size
Comet processes data in columnar batches. The batch size is controlled by `spark.comet.batchSize` (default
diff --git a/spark/src/main/scala/org/apache/comet/CometConf.scala b/spark/src/main/scala/org/apache/comet/CometConf.scala
index 59d22d887f..a570b03325 100644
--- a/spark/src/main/scala/org/apache/comet/CometConf.scala
+++ b/spark/src/main/scala/org/apache/comet/CometConf.scala
@@ -369,10 +369,14 @@ object CometConf extends ShimCometConf {
"difference between the first two is native memory that the pools are not accounting " +
"for. The executor logs one line per interval however many tasks are " +
"running, and one more after the last plan finishes. It logs a warning when the " +
- "native memory looks larger than the executor's container allows. This is an executor " +
- "setting, read when an executor starts its first Comet native plan, so it must be set " +
- "when the application is submitted. An invalid value disables the log with a warning. " +
- s"Set to 0 to disable. $TUNING_GUIDE.")
+ "native memory looks larger than the executor's container allows. When " +
+ "spark.eventLog.enabled is true and the application runs the Comet plugin, the " +
+ "executor also sends its samples to the driver, which writes the one with the most " +
+ "untracked memory and the last of every minute to the event log, as " +
+ "CometExecutorMemoryUsage events. This is an " +
+ "executor setting, read when an executor starts its first Comet native plan, so it " +
+ "must be set when the application is submitted. An invalid value disables the log " +
+ s"with a warning. Set to 0 to disable. $TUNING_GUIDE.")
.timeConf(TimeUnit.MILLISECONDS)
.checkValue(_ >= 0, "The memory usage log interval must not be negative")
.createWithDefault(TimeUnit.SECONDS.toMillis(10))
diff --git a/spark/src/main/scala/org/apache/comet/CometExecIterator.scala b/spark/src/main/scala/org/apache/comet/CometExecIterator.scala
index b187cbd0c1..69b12ee12e 100644
--- a/spark/src/main/scala/org/apache/comet/CometExecIterator.scala
+++ b/spark/src/main/scala/org/apache/comet/CometExecIterator.scala
@@ -423,6 +423,14 @@ object CometExecIterator extends Logging {
*/
private var limitExceededAtLastLog = false
+ /**
+ * Whether the memory usage log failed to send a sample to the driver, after which it stops
+ * sending them. Only the log's own thread sets it.
+ */
+ private var eventLogSendFailed = false
+
+ private val EVENT_LOG_SUMMARY_INTERVAL_NS = TimeUnit.MINUTES.toNanos(1)
+
/**
* Starts the executor's native memory usage log when the first native plan is created, unless
* `spark.comet.memory.logInterval` is 0.
@@ -434,6 +442,12 @@ object CometExecIterator extends Logging {
* whole run inside one `executePlan` call: a plan rooted at a native shuffle writer consumes
* all of its input before it returns, and a plan fed directly by native scans parks the task
* thread until its next batch is ready.
+ *
+ * When the application writes an event log and runs the Comet plugin, each sample also goes to
+ * the driver, which writes a summary of them to the event log; see [[sendToEventLog]].
+ * Executors' own logs are spread over the cluster and often go away with their containers,
+ * while the event log keeps every executor's samples in one place, next to the jobs and stages
+ * they ran alongside.
*/
private def startMemoryUsageLog(): Unit = {
if (memoryUsageLogStarted.compareAndSet(false, true)) {
@@ -588,7 +602,10 @@ object CometExecIterator extends Logging {
try {
val usage = nativeLib.getMemoryUsage()
val jvmArrow = JvmArrowMemory.current()
- memoryUsageMessage(usage, jvmArrow, plansAtLastMemoryUsageLog).foreach(logInfo(_))
+ memoryUsageMessage(usage, jvmArrow, plansAtLastMemoryUsageLog).foreach { message =>
+ logInfo(message)
+ sendToEventLog(usage, jvmArrow)
+ }
plansAtLastMemoryUsageLog = usage(3)
val warning = limitBytes.flatMap(
nativeMemoryLimitWarning(usage, jvmArrow, CometTaskMemoryManager.sparkOffHeapUsed(), _))
@@ -607,6 +624,102 @@ object CometExecIterator extends Logging {
}
}
+ /**
+ * Sends a memory usage sample to the driver, which writes a summary of the samples to the event
+ * log; see [[MemoryUsageSummary]]. The executor plugin is looked up for every sample rather
+ * than once, because in local mode each SparkContext starts its own. Executors receive the
+ * driver's `spark.*` settings, so the plugin sees whether the driver writes an event log. The
+ * send is one-way, so Spark reports a failure to reach a remote driver in its own log. A
+ * failure that surfaces here, such as a driver endpoint that could not be found, persists, so
+ * the log stops sending after it.
+ */
+ private[apache] def sendToEventLog(usage: Array[Long], jvmArrow: JvmArrowMemory): Unit =
+ if (!eventLogSendFailed) {
+ CometExecutorPlugin.eventLogContext.foreach { pluginContext =>
+ try {
+ pluginContext.send(
+ memoryUsageEvent(
+ pluginContext.executorID(),
+ System.currentTimeMillis(),
+ usage,
+ jvmArrow))
+ } catch {
+ case NonFatal(e) =>
+ logWarning(
+ "Not sending native memory usage to the event log from now on: could not send " +
+ "it to the driver",
+ e)
+ eventLogSendFailed = true
+ }
+ }
+ }
+
+ /**
+ * The memory usage samples that one executor sent the driver since the event log last recorded
+ * any of them. The driver writes, and flushes, each event it records, so rather than one event
+ * per executor per interval it records two samples for each minute an executor reports: the one
+ * with the most untracked memory, and the last, which is the one after the last plan finishes
+ * when the executor goes idle. Spark's executor metrics reach the event log as peaks in the
+ * same way. The driver holds the summary rather than the executor: an executor that the
+ * application stops, or the cluster manager kills, has no chance to send what it holds.
+ */
+ class MemoryUsageSummary {
+
+ /** The sample with the most untracked memory since the previous summary, and that memory. */
+ private var peak: Option[(CometExecutorMemoryUsage, Long)] = None
+
+ private var last: Option[CometExecutorMemoryUsage] = None
+
+ /**
+ * When the driver received the first sample since the previous summary, in nanoseconds by its
+ * own monotonic clock, rather than when the executor took it by a clock that can differ.
+ */
+ private var openedAt = 0L
+
+ /** Adds `sample`, which the driver received at `receivedAt`, by its monotonic clock. */
+ def add(sample: CometExecutorMemoryUsage, receivedAt: Long): Unit = {
+ if (last.isEmpty) openedAt = receivedAt
+ val jvmArrow = JvmArrowMemory(sample.jvmArrowAllocated, sample.jvmArrowImported)
+ val untracked = untrackedMemory(sample.nativeAllocated, sample.poolsReserved, jvmArrow)
+ if (peak.forall(_._2 < untracked)) peak = Some((sample, untracked))
+ last = Some(sample)
+ }
+
+ /**
+ * The samples to record, if the first of them arrived a minute or more before `now`, by the
+ * same clock as `add`, after which the summary starts over, or nothing.
+ */
+ def flushIfDue(now: Long): Seq[CometExecutorMemoryUsage] =
+ if (last.nonEmpty && now - openedAt >= EVENT_LOG_SUMMARY_INTERVAL_NS) flush() else Nil
+
+ /** The samples to record since the previous summary, after which the summary starts over. */
+ def flush(): Seq[CometExecutorMemoryUsage] = {
+ val summary = (peak.map(_._1).toSeq ++ last).distinct
+ peak = None
+ last = None
+ summary
+ }
+ }
+
+ /**
+ * The event log's record of the memory usage sample `usage`, as returned by
+ * [[Native.getMemoryUsage]], and `jvmArrow`, which `executorId` took at `time`.
+ */
+ def memoryUsageEvent(
+ executorId: String,
+ time: Long,
+ usage: Array[Long],
+ jvmArrow: JvmArrowMemory): CometExecutorMemoryUsage =
+ CometExecutorMemoryUsage(
+ executorId,
+ time,
+ nativeAllocated = usage(0),
+ poolsReserved = usage(1),
+ pools = usage(2),
+ plans = usage(3),
+ jvmArrowAllocated = jvmArrow.allocated,
+ jvmArrowImported = jvmArrow.imported)
+
/**
* The Arrow memory Comet holds on the JVM side, read from its allocators' own accounting:
* `allocated` for the whole `CometArrowAllocator` tree, and `imported` for the part of it
@@ -641,9 +754,9 @@ object CometExecIterator extends Logging {
/**
* The memory usage log line for `usage`, as returned by [[Native.getMemoryUsage]], and
- * `jvmArrow`, or None to stay quiet. The log reports while native plans are running, and once
- * more after the last of them finishes, so that allocation that outlives them is visible, then
- * waits for plans to run again.
+ * `jvmArrow`, or None to stay quiet, in the executor's log and in the event log alike. The log
+ * reports while native plans are running, and once more after the last of them finishes, so
+ * that allocation that outlives them is visible, then waits for plans to run again.
*/
def memoryUsageMessage(
usage: Array[Long],
@@ -660,6 +773,14 @@ object CometExecIterator extends Logging {
}
}
+ /**
+ * The memory Comet holds outside the JVM heap that Spark's off-heap pool does not account for:
+ * `allocated` native memory, less the `reserved` part, plus the Arrow memory the JVM allocated
+ * itself; see [[nativeMemoryLimitWarning]].
+ */
+ def untrackedMemory(allocated: Long, reserved: Long, jvmArrow: JvmArrowMemory): Long =
+ math.max(allocated + jvmArrow.allocatedByJvm - reserved, 0L)
+
/**
* A warning if the executor's native footprint exceeds `limitBytes`, the container's memory
* outside the JVM heap; see [[nativeMemoryLimit]].
@@ -683,7 +804,7 @@ object CometExecIterator extends Logging {
jvmArrow: JvmArrowMemory,
sparkOffHeapUsed: Long,
limitBytes: Long): Option[String] = {
- val untracked = math.max(usage(0) + jvmArrow.allocatedByJvm - usage(1), 0L)
+ val untracked = untrackedMemory(usage(0), usage(1), jvmArrow)
val footprint = untracked + sparkOffHeapUsed
if (footprint > limitBytes) {
Some(s"Comet memory that Spark does not account for (${toMiB(untracked)}, native and JVM " +
diff --git a/spark/src/main/scala/org/apache/comet/CometExecutorMemoryUsage.scala b/spark/src/main/scala/org/apache/comet/CometExecutorMemoryUsage.scala
new file mode 100644
index 0000000000..fae95302e7
--- /dev/null
+++ b/spark/src/main/scala/org/apache/comet/CometExecutorMemoryUsage.scala
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet
+
+import org.apache.spark.scheduler.SparkListenerEvent
+
+/**
+ * One sample of an executor's native memory usage log, in the form the event log records it. When
+ * the application writes an event log and runs the Comet plugin, the executor sends each sample
+ * to the driver plugin as well as logging it, and the driver posts a summary of them to the
+ * listener bus, which writes each sample of it to the event log as JSON with `"Event"` set to
+ * this class's name. See `CometExecIterator.MemoryUsageSummary`.
+ *
+ * The memory figures are in bytes and are the ones the log line reports; see
+ * `CometExecIterator.memoryUsageMessage`.
+ *
+ * @param executorId
+ * The executor that took the sample, `driver` in local mode.
+ * @param time
+ * When the executor took the sample, in milliseconds since the epoch by the executor's clock.
+ * @param nativeAllocated
+ * The memory that Comet's native code has allocated and not yet freed, whether or not a memory
+ * pool tracks it.
+ * @param poolsReserved
+ * The memory reserved across Comet's memory pools, counting a pool shared by several plans
+ * once, less any that a pool recorded beyond what Spark granted it.
+ * @param pools
+ * The number of live memory pools.
+ * @param plans
+ * The number of native plans created and not yet released.
+ * @param jvmArrowAllocated
+ * The Arrow memory Comet holds on the JVM side.
+ * @param jvmArrowImported
+ * The part of `jvmArrowAllocated` imported from native code, which `nativeAllocated` already
+ * counts.
+ */
+case class CometExecutorMemoryUsage(
+ executorId: String,
+ time: Long,
+ nativeAllocated: Long,
+ poolsReserved: Long,
+ pools: Long,
+ plans: Long,
+ jvmArrowAllocated: Long,
+ jvmArrowImported: Long)
+ extends SparkListenerEvent
diff --git a/spark/src/main/scala/org/apache/spark/Plugins.scala b/spark/src/main/scala/org/apache/spark/Plugins.scala
index ebabd77340..c3246679d0 100644
--- a/spark/src/main/scala/org/apache/spark/Plugins.scala
+++ b/spark/src/main/scala/org/apache/spark/Plugins.scala
@@ -21,17 +21,22 @@ package org.apache.spark
import java.{util => ju}
import java.util.Collections
+import java.util.concurrent.atomic.AtomicReference
+import scala.collection.mutable
import scala.util.Try
import org.apache.spark.api.plugin.{DriverPlugin, ExecutorPlugin, PluginContext, SparkPlugin}
import org.apache.spark.internal.Logging
-import org.apache.spark.internal.config.{EXECUTOR_MEMORY_OVERHEAD, EXECUTOR_MEMORY_OVERHEAD_FACTOR}
+import org.apache.spark.internal.config.{EVENT_LOG_ENABLED, EXECUTOR_MEMORY_OVERHEAD, EXECUTOR_MEMORY_OVERHEAD_FACTOR}
+import org.apache.spark.scheduler.{SparkListener, SparkListenerApplicationEnd, SparkListenerExecutorMetricsUpdate, SparkListenerExecutorRemoved}
import org.apache.spark.sql.internal.StaticSQLConf
+import org.apache.spark.util.{Clock, SystemClock}
-import org.apache.comet.{COMET_VERSION, CometExecIterator, CometSparkSessionExtensions, NativeBase}
+import org.apache.comet.{COMET_VERSION, CometExecIterator, CometExecutorMemoryUsage, CometSparkSessionExtensions, NativeBase}
import org.apache.comet.{CometConf, ConfigEntry}
import org.apache.comet.CometConf.{COMET_ICEBERG_WRITE_REPORT_DIR, COMET_METRICS_ENABLED, COMET_ONHEAP_ENABLED}
+import org.apache.comet.CometExecIterator.MemoryUsageSummary
import org.apache.comet.CometKryoRegistrator
import org.apache.comet.annotation.Public
import org.apache.comet.iceberg.IcebergWriteReportListener
@@ -47,11 +52,49 @@ import org.apache.comet.iceberg.IcebergWriteReportListener
*
* To enable this plugin, set the config "spark.plugins" to `org.apache.spark.CometPlugin`.
*/
-class CometDriverPlugin extends DriverPlugin with Logging {
+class CometDriverPlugin private[spark] (clock: Clock) extends DriverPlugin with Logging {
+
+ def this() = this(new SystemClock())
+
+ // Set by init, before Spark delivers any message, and read on the RPC thread that delivers them.
+ @volatile private var sparkContext: SparkContext = _
+
+ // By executor, the memory usage samples that the event log has yet to record. The RPC thread
+ // that delivers samples shares it with the threads that record them.
+ private val memoryUsageSummaries = mutable.HashMap.empty[String, MemoryUsageSummary]
override def init(sc: SparkContext, pluginContext: PluginContext): ju.Map[String, String] = {
logInfo("CometDriverPlugin init")
+ sparkContext = sc
+ if (sc.conf.get(EVENT_LOG_ENABLED)) {
+ // A queue of its own, so that a slow listener on the shared queue cannot hold the
+ // application's end back until the listener bus has stopped, which drops what is posted
+ // after it.
+ sc.listenerBus.addToQueue(
+ new SparkListener {
+ // Every executor heartbeat posts one, whether or not the executor is busy, so this ends
+ // an idle executor's summary too. The event log does not record the heartbeat itself.
+ override def onExecutorMetricsUpdate(event: SparkListenerExecutorMetricsUpdate): Unit =
+ recordMemoryUsage(memoryUsageSummaries.synchronized {
+ memoryUsageSummaries
+ .get(event.execId)
+ .toList
+ .flatMap(_.flushIfDue(clock.nanoTime()))
+ })
+
+ // An executor that has gone away sends no more heartbeats to end its summary.
+ override def onExecutorRemoved(event: SparkListenerExecutorRemoved): Unit =
+ recordMemoryUsage(memoryUsageSummaries.synchronized {
+ memoryUsageSummaries.remove(event.executorId).toList.flatMap(_.flush())
+ })
+
+ override def onApplicationEnd(event: SparkListenerApplicationEnd): Unit =
+ recordRemainingMemoryUsage()
+ },
+ "comet")
+ }
+
// Expose the Comet build version as a Spark config so it can be queried at runtime, e.g.
// `spark.conf.get("spark.comet.version")` or `SET spark.comet.version` in SQL. This is set
// before the off-heap check below so the version is reported even when Comet is otherwise
@@ -82,16 +125,42 @@ class CometDriverPlugin extends DriverPlugin with Logging {
extraConfs
}
- override def receive(message: Any): AnyRef = super.receive(message)
+ override def receive(message: Any): AnyRef = message match {
+ // An executor's memory usage sample. A one-way message gets no reply, and Spark logs any
+ // reply that is not null.
+ case sample: CometExecutorMemoryUsage =>
+ memoryUsageSummaries.synchronized {
+ memoryUsageSummaries
+ .getOrElseUpdate(sample.executorId, new MemoryUsageSummary)
+ .add(sample, clock.nanoTime())
+ }
+ null
+ case _ => super.receive(message)
+ }
override def shutdown(): Unit = {
logInfo("CometDriverPlugin shutdown")
+ // From Spark 4.0 the listener bus stops after the plugins, so this records what is left even
+ // if the listener has yet to see the application end. Before 4.0 the bus has stopped already.
+ recordRemainingMemoryUsage()
+
NativeBase.releaseNative()
super.shutdown()
}
+ // Posting samples to the listener bus is what writes them to the event log.
+ private def recordMemoryUsage(samples: Seq[CometExecutorMemoryUsage]): Unit =
+ samples.foreach(sparkContext.listenerBus.post)
+
+ private def recordRemainingMemoryUsage(): Unit =
+ recordMemoryUsage(memoryUsageSummaries.synchronized {
+ val remaining = memoryUsageSummaries.values.flatMap(_.flush()).toList
+ memoryUsageSummaries.clear()
+ remaining
+ })
+
override def registerMetrics(appId: String, pluginContext: PluginContext): Unit =
super.registerMetrics(appId, pluginContext)
@@ -290,15 +359,24 @@ object CometDriverPlugin extends Logging {
class CometExecutorPlugin extends ExecutorPlugin with Logging {
+ private var context: PluginContext = _
+
override def init(ctx: PluginContext, extraConf: ju.Map[String, String]): Unit = {
logInfo("CometExecutorPlugin init")
+ context = ctx
+ CometExecutorPlugin.current.set(ctx)
+
super.init(ctx, extraConf)
}
override def shutdown(): Unit = {
logInfo("CometExecutorPlugin shutdown")
+ // Unless a later plugin in the same JVM, which local mode starts for each SparkContext, has
+ // already replaced it.
+ CometExecutorPlugin.current.compareAndSet(context, null)
+
NativeBase.releaseNative()
super.shutdown()
@@ -306,6 +384,20 @@ class CometExecutorPlugin extends ExecutorPlugin with Logging {
}
+object CometExecutorPlugin {
+
+ private val current = new AtomicReference[PluginContext]()
+
+ /**
+ * The context of the executor plugin running in this JVM, through which the executor sends its
+ * memory usage samples to the driver plugin when the application writes an event log. None
+ * without the Comet plugin or an event log, and after the executor has shut the plugin down.
+ * The flag is read as the driver reads it, which ignores surrounding whitespace.
+ */
+ private[apache] def eventLogContext: Option[PluginContext] =
+ Option(current.get()).filter(_.conf.get(EVENT_LOG_ENABLED))
+}
+
/**
* The Comet plugin for Spark. To enable this plugin, set the config "spark.plugins" to
* `org.apache.spark.CometPlugin`
diff --git a/spark/src/test/scala/org/apache/spark/CometExecIteratorLifecycleSuite.scala b/spark/src/test/scala/org/apache/spark/CometExecIteratorLifecycleSuite.scala
index 127d9fc9cd..80f8c75c78 100644
--- a/spark/src/test/scala/org/apache/spark/CometExecIteratorLifecycleSuite.scala
+++ b/spark/src/test/scala/org/apache/spark/CometExecIteratorLifecycleSuite.scala
@@ -22,6 +22,7 @@ package org.apache.spark
import java.io.ByteArrayInputStream
import java.lang.ref.WeakReference
import java.util.Properties
+import java.util.concurrent.TimeUnit
import java.util.concurrent.atomic.{AtomicBoolean, AtomicInteger}
import scala.reflect.ClassTag
@@ -39,8 +40,9 @@ import org.apache.spark.sql.comet.{CometExec, CometExecUtils, CometMetricNode, C
import org.apache.spark.sql.comet.execution.arrow.CometArrowStream
import org.apache.spark.sql.functions.{col, udf}
import org.apache.spark.sql.types.{LongType, StructField, StructType}
+import org.apache.spark.util.JsonProtocol
-import org.apache.comet.{CometConf, CometExecIterator, CometNativeException, CometShuffleBlockIterator, Native}
+import org.apache.comet.{CometConf, CometExecIterator, CometExecutorMemoryUsage, CometNativeException, CometShuffleBlockIterator, Native}
import org.apache.comet.serde.Config.ConfigMap
import org.apache.comet.serde.OperatorOuterClass
@@ -475,6 +477,72 @@ class CometExecIteratorLifecycleSuite extends CometTestBase {
assert(CometExecIterator.memoryUsageMessage(idle, noArrow, plansAtLastLog = 0).isEmpty)
}
+ test("the memory usage log's event log record reads back from the event log's JSON") {
+ import CometExecIterator.{memoryUsageEvent, JvmArrowMemory}
+ val mib = 1024L * 1024
+ val event = memoryUsageEvent(
+ executorId = "7",
+ time = 1700000000000L,
+ usage = Array(300 * mib, 100 * mib, 2L, 3L),
+ jvmArrow = JvmArrowMemory(allocated = 40 * mib, imported = 10 * mib))
+ assert(
+ event == CometExecutorMemoryUsage(
+ executorId = "7",
+ time = 1700000000000L,
+ nativeAllocated = 300 * mib,
+ poolsReserved = 100 * mib,
+ pools = 2L,
+ plans = 3L,
+ jvmArrowAllocated = 40 * mib,
+ jvmArrowImported = 10 * mib))
+
+ // The event log writes an event it has no format of its own for with Jackson, under its class
+ // name, which is how the history server reads it back, or skips it when Comet is not on its
+ // classpath.
+ val json = JsonProtocol.sparkEventToJsonString(event)
+ assert(json.contains(s""""Event":"${classOf[CometExecutorMemoryUsage].getName}""""), json)
+ assert(json.contains(s""""poolsReserved":${100 * mib}"""), json)
+ assert(JsonProtocol.sparkEventFromJson(json) == event)
+ }
+
+ test("the event log records the peak and the last memory usage sample of each minute") {
+ import CometExecIterator.{memoryUsageEvent, JvmArrowMemory, MemoryUsageSummary}
+ val mib = 1024L * 1024
+ // A sample taken `seconds` in, with `untrackedMiB` of native memory that no pool reserves.
+ def sample(
+ seconds: Int,
+ untrackedMiB: Long,
+ jvmArrow: JvmArrowMemory = JvmArrowMemory(0L, 0L)) =
+ memoryUsageEvent(
+ executorId = "7",
+ time = 1700000000000L + seconds * 1000L,
+ usage = Array((100 + untrackedMiB) * mib, 100 * mib, 2L, 3L),
+ jvmArrow = jvmArrow)
+
+ val summary = new MemoryUsageSummary
+ val peak = sample(10, 50)
+ val last = sample(50, 20)
+ // What decides is when the driver received the samples, by its own monotonic clock in
+ // nanoseconds, not the times the executor took them.
+ val second = TimeUnit.SECONDS.toNanos(1)
+ summary.add(sample(0, 10), receivedAt = 0L)
+ summary.add(peak, receivedAt = 10 * second)
+ summary.add(last, receivedAt = 50 * second)
+ // The summary ends a minute after the driver received its first sample.
+ assert(summary.flushIfDue(60 * second - 1).isEmpty)
+ assert(summary.flushIfDue(60 * second) == Seq(peak, last))
+
+ // Arrow memory that the JVM allocated counts as untracked, and the part imported from native
+ // does not, since the native figure already counts it.
+ val imported = sample(70, 0, JvmArrowMemory(allocated = 900 * mib, imported = 900 * mib))
+ val jvm = sample(80, 0, JvmArrowMemory(allocated = 40 * mib, imported = 0L))
+ Seq(sample(60, 30), imported, jvm).foreach(summary.add(_, receivedAt = 70 * second))
+ // What is left when the executor goes away or the application stops, a peak that is also the
+ // last sample recorded once.
+ assert(summary.flush() == Seq(jvm))
+ assert(summary.flush().isEmpty)
+ }
+
test("the memory usage log reads JVM Arrow memory from the allocators, imports apart") {
import CometExecIterator.JvmArrowMemory
val root = new RootAllocator(Long.MaxValue)
diff --git a/spark/src/test/scala/org/apache/spark/CometPluginsSuite.scala b/spark/src/test/scala/org/apache/spark/CometPluginsSuite.scala
index 25deaf99f9..f85045a57e 100644
--- a/spark/src/test/scala/org/apache/spark/CometPluginsSuite.scala
+++ b/spark/src/test/scala/org/apache/spark/CometPluginsSuite.scala
@@ -20,14 +20,20 @@
package org.apache.spark
import java.io.File
+import java.nio.charset.StandardCharsets
+import java.nio.file.Files
+import java.util.concurrent.TimeUnit
import org.apache.logging.log4j.Level
+import org.apache.spark.api.plugin.{DriverPlugin, ExecutorPlugin, PluginContext, SparkPlugin}
+import org.apache.spark.scheduler.{SparkListenerApplicationEnd, SparkListenerEvent, SparkListenerExecutorMetricsUpdate, SparkListenerExecutorRemoved}
import org.apache.spark.sql.{CometTestBase, SaveMode, SparkSession}
import org.apache.spark.sql.comet.CometPlan
import org.apache.spark.sql.comet.execution.shuffle.{CometShuffleExchangeExec, CometShuffleManager}
import org.apache.spark.sql.internal.StaticSQLConf
+import org.apache.spark.util.{JsonProtocol, ManualClock, Utils}
-import org.apache.comet.{COMET_VERSION, CometConf}
+import org.apache.comet.{COMET_VERSION, CometConf, CometExecIterator, CometExecutorMemoryUsage}
class CometPluginsSuite extends CometTestBase {
override protected def sparkConf: SparkConf = {
@@ -162,6 +168,11 @@ class CometPluginsSuite extends CometTestBase {
assert(execMemOverhead3 == "2G")
assert(execMemOverhead4 == "2G")
}
+
+ test("the memory usage log sends the driver nothing without an event log") {
+ // The suite runs the Comet plugin but writes no event log.
+ assert(CometExecutorPlugin.eventLogContext.isEmpty)
+ }
}
class CometPluginsDefaultSuite extends CometTestBase {
@@ -418,3 +429,135 @@ class CometPluginsSparkShuffleManagerSuite extends CometTestBase {
}
}
}
+
+class CometPluginsEventLogSuite extends SparkFunSuite {
+
+ import CometExecIterator.{memoryUsageEvent, JvmArrowMemory}
+
+ /**
+ * Runs `f` in an application that runs `plugin` and writes an event log, passing it the context
+ * through which the executor sends memory usage samples, and returns the event log's events
+ * once the application has stopped.
+ */
+ private def eventLog(plugin: Class[_ <: SparkPlugin] = classOf[CometPlugin])(
+ f: (SparkContext, PluginContext) => Unit): Seq[SparkListenerEvent] = {
+ val eventLogDir = Utils.createTempDir()
+ try {
+ val sc = new SparkContext(
+ new SparkConf()
+ .setMaster("local[1]")
+ .setAppName(getClass.getSimpleName)
+ .set("spark.plugins", plugin.getName)
+ .set("spark.eventLog.enabled", "true")
+ .set("spark.eventLog.dir", eventLogDir.toURI.toString)
+ // One plain file, which the test reads directly. Spark 4 compresses and rolls it by
+ // default.
+ .set("spark.eventLog.compress", "false")
+ .set("spark.eventLog.rolling.enabled", "false"))
+ try {
+ val pluginContext = CometExecutorPlugin.eventLogContext
+ assert(pluginContext.isDefined, "The executor should send samples to the event log")
+ f(sc, pluginContext.get)
+ } finally {
+ sc.stop()
+ }
+ eventLogDir
+ .listFiles()
+ .toSeq
+ .filter(file => file.isFile && !file.getName.startsWith("."))
+ .flatMap(file =>
+ new String(Files.readAllBytes(file.toPath), StandardCharsets.UTF_8)
+ .split("\n")
+ .map(JsonProtocol.sparkEventFromJson))
+ } finally {
+ Utils.deleteRecursively(eventLogDir)
+ }
+ }
+
+ private val marker =
+ memoryUsageEvent("marker", 0L, Array(0L, 0L, 0L, 0L), JvmArrowMemory(0L, 0L))
+
+ /**
+ * Posts `event`, and a marker once every listener has handled it. The bus hands an event to one
+ * queue at a time, so what the driver plugin's listener records on `event` can reach the event
+ * log ahead of `event` itself. It always reaches it ahead of the marker.
+ */
+ private def postAndMark(sc: SparkContext, event: SparkListenerEvent): Unit = {
+ sc.listenerBus.post(event)
+ sc.listenerBus.waitUntilEmpty()
+ sc.listenerBus.post(marker)
+ }
+
+ /**
+ * Asserts that the event log records exactly one sample `isSample` accepts, ahead of the marker
+ * and so not at the application's end.
+ */
+ private def assertRecordedOnceBeforeMarker(events: Seq[SparkListenerEvent])(
+ isSample: SparkListenerEvent => Boolean): Unit = {
+ val recorded = events.indices.filter(i => isSample(events(i)))
+ assert(recorded.size == 1 && recorded(0) < events.indexOf(marker), events)
+ }
+
+ test("the driver records what the memory usage log sent it when the application ends") {
+ // An allocation no real sample has, to tell this sample apart from any that the memory usage
+ // log running in this JVM sends, and large enough to be the most untracked memory of its
+ // summary.
+ val usage = Array(123456789L * 1024, 0L, 1L, 1L)
+ val jvmArrow = JvmArrowMemory(allocated = 3456789L, imported = 456789L)
+ val events = eventLog() { (sc, pluginContext) =>
+ CometExecIterator.sendToEventLog(usage, jvmArrow)
+ // The driver plugin receives its messages in order, and `ask`, unlike the log's one-way
+ // `send`, returns once it has received this one, so it has the sample above by then. It
+ // replies to none of them.
+ assert(
+ pluginContext.ask(
+ memoryUsageEvent("barrier", 0L, Array(0L, 0L, 0L, 0L), jvmArrow)) == null)
+ // The end as the listener sees it, which alone records what the driver holds on Spark 3.4
+ // and 3.5. From Spark 4.0 the plugin's shutdown would record it too, when the context stops.
+ postAndMark(sc, SparkListenerApplicationEnd(System.currentTimeMillis()))
+ }
+ assertRecordedOnceBeforeMarker(events) {
+ // Local mode runs the executor inside the driver.
+ case sample: CometExecutorMemoryUsage =>
+ sample == memoryUsageEvent(SparkContext.DRIVER_IDENTIFIER, sample.time, usage, jvmArrow)
+ case _ => false
+ }
+ }
+
+ test("the driver records what an executor sent since its last summary when it goes away") {
+ val sample = memoryUsageEvent(
+ "lost",
+ System.currentTimeMillis(),
+ Array(300L * 1024 * 1024, 100L * 1024 * 1024, 2L, 3L),
+ JvmArrowMemory(0L, 0L))
+ val events = eventLog() { (sc, pluginContext) =>
+ pluginContext.ask(sample)
+ postAndMark(sc, SparkListenerExecutorRemoved(sample.time, sample.executorId, "test"))
+ }
+ assertRecordedOnceBeforeMarker(events)(_ == sample)
+ }
+
+ test("the driver records an idle executor's samples at its first heartbeat a minute later") {
+ val sample = memoryUsageEvent(
+ "idle",
+ System.currentTimeMillis(),
+ Array(300L * 1024 * 1024, 100L * 1024 * 1024, 2L, 0L),
+ JvmArrowMemory(0L, 0L))
+ val events = eventLog(classOf[ManualClockCometPlugin]) { (sc, pluginContext) =>
+ pluginContext.ask(sample)
+ ManualClockCometPlugin.clock.advance(TimeUnit.MINUTES.toMillis(1))
+ postAndMark(sc, SparkListenerExecutorMetricsUpdate(sample.executorId, Seq.empty))
+ }
+ assertRecordedOnceBeforeMarker(events)(_ == sample)
+ }
+}
+
+/** The Comet plugin with a driver clock that `CometPluginsEventLogSuite` moves by hand. */
+class ManualClockCometPlugin extends SparkPlugin {
+ override def driverPlugin(): DriverPlugin = new CometDriverPlugin(ManualClockCometPlugin.clock)
+ override def executorPlugin(): ExecutorPlugin = new CometExecutorPlugin
+}
+
+object ManualClockCometPlugin {
+ val clock = new ManualClock()
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org