You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "LuciferYang (via GitHub)" <gi...@apache.org> on 2026/07/24 08:57:39 UTC
[PR] [GLUTEN-12621][CORE] Fix NoSuchElementException in untracked memory mode without off-heap size [gluten]
LuciferYang opened a new pull request, #12622:
URL: https://github.com/apache/gluten/pull/12622
### What changes were proposed in this pull request?
`GlutenDriverPlugin.setPredefinedConfigs` reads the off-heap size in its non-dynamic-sizing branch with the single-arg `conf.getSizeAsBytes(SPARK_OFFHEAP_SIZE_KEY)`. `checkOffHeapSettings` returns early for untracked memory mode (`spark.gluten.memory.untracked=true`) and skips the requirement that the size be set, so in that mode the key can be absent. `SparkConf.get(String)` throws `NoSuchElementException` on a missing key, so driver plugin init crashes instead of proceeding under untracked semantics.
The same single-arg read exists on the Velox driver-start path: `VeloxListenerApi.onDriverStart` also calls `conf.getSizeAsBytes(SPARK_OFFHEAP_SIZE_KEY)`, and it runs after `setPredefinedConfigs` (via `init` calling `Component.onDriverStart`). The dynamic-sizing branch already shields that reader by doing `conf.set(SPARK_OFFHEAP_SIZE_KEY, "0")`; untracked mode did not, so it was the one unshielded path.
This change normalizes the key in the untracked branch: when `spark.memory.offHeap.size` is absent, set it to `0`, the same way the dynamic-sizing branch does. One change covers both readers. Normal mode is unaffected because `checkOffHeapSettings` already guarantees the key is set there.
### How was this patch tested?
Added `GlutenDriverPluginSuite` with two tests: untracked mode without an off-heap size no longer throws and yields a 0 off-heap budget, and the normal path reads a configured `512m`. The untracked test also asserts `spark.memory.offHeap.size` is readable afterward, which is what the downstream `VeloxListenerApi` path needs. The first test fails on the current code (it throws `NoSuchElementException`) and passes after the fix.
Closes #12621
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12621][CORE] Fix NoSuchElementException in untracked memory mode without off-heap size [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12622:
URL: https://github.com/apache/gluten/pull/12622#issuecomment-5068090750
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12621][CORE] Fix NoSuchElementException in untracked memory mode without off-heap size [gluten]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch merged PR #12622:
URL: https://github.com/apache/gluten/pull/12622
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12621][CORE] Fix NoSuchElementException in untracked memory mode without off-heap size [gluten]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #12622:
URL: https://github.com/apache/gluten/pull/12622#discussion_r3646102573
##########
gluten-core/src/main/scala/org/apache/gluten/GlutenPlugin.scala:
##########
@@ -134,6 +135,15 @@ private object GlutenDriverPlugin extends Logging {
((onHeapSize - (300 * 1024 * 1024)) *
conf.getDouble(GlutenCoreConfig.DYNAMIC_OFFHEAP_SIZING_MEMORY_FRACTION.key, 0.6d)).toLong
} else {
+ // Untracked memory mode skips the off-heap size requirement in checkOffHeapSettings, so
+ // the key may be absent here. Normalize it to 0 (mirroring the dynamic-sizing branch
+ // above) so downstream readers that read spark.memory.offHeap.size directly, e.g.
+ // VeloxListenerApi.onDriverStart, don't hit NoSuchElementException. Normal mode always has
+ // the key set because checkOffHeapSettings enforced it, so this only affects untracked
+ // mode without an explicit off-heap size.
+ if (!conf.contains(GlutenCoreConfig.SPARK_OFFHEAP_SIZE_KEY)) {
Review Comment:
It's still needed for untracked mode. You're right that we validate the off-heap size in `checkOffHeapSettings`, but that path only runs in normal mode: untracked mode returns early at L86-89 (the `COLUMNAR_MEMORY_UNTRACKED` branch), before the L92-102 size check, so `spark.memory.offHeap.size` can stay absent.
Repro: `spark.gluten.memory.untracked=true` with `spark.memory.offHeap.size` unset and dynamic sizing off → `checkOffHeapSettings` returns at L89 → the else-branch (main L136) calls the single-arg `conf.getSizeAsBytes(SPARK_OFFHEAP_SIZE_KEY)`, which throws `NoSuchElementException` because `SparkConf.get(String)` reads the raw settings map and doesn't fall back to the ConfigEntry default. The added `GlutenDriverPluginSuite` test "setPredefinedConfigs does not throw in untracked mode without an off-heap size" fails without this change.
If it reads clearer, I can add a short comment at the untracked early-return noting the size isn't validated there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12621][CORE] Fix NoSuchElementException in untracked memory mode without off-heap size [gluten]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #12622:
URL: https://github.com/apache/gluten/pull/12622#discussion_r3654599439
##########
gluten-core/src/main/scala/org/apache/gluten/GlutenPlugin.scala:
##########
@@ -134,6 +135,15 @@ private object GlutenDriverPlugin extends Logging {
((onHeapSize - (300 * 1024 * 1024)) *
conf.getDouble(GlutenCoreConfig.DYNAMIC_OFFHEAP_SIZING_MEMORY_FRACTION.key, 0.6d)).toLong
} else {
+ // Untracked memory mode skips the off-heap size requirement in checkOffHeapSettings, so
+ // the key may be absent here. Normalize it to 0 (mirroring the dynamic-sizing branch
+ // above) so downstream readers that read spark.memory.offHeap.size directly, e.g.
+ // VeloxListenerApi.onDriverStart, don't hit NoSuchElementException. Normal mode always has
+ // the key set because checkOffHeapSettings enforced it, so this only affects untracked
+ // mode without an explicit off-heap size.
+ if (!conf.contains(GlutenCoreConfig.SPARK_OFFHEAP_SIZE_KEY)) {
Review Comment:
To confirm this reproduces on the real driver-init path and not only in the unit test:
With `spark.plugins=org.apache.gluten.GlutenPlugin`, `spark.gluten.memory.untracked=true`, and `spark.memory.offHeap.size` left unset, constructing a `SparkContext` runs `GlutenDriverPlugin.init` -> `setPredefinedConfigs` and fails with:
```
java.util.NoSuchElementException: spark.memory.offHeap.size
```
I checked this locally by wiring a test through `new SparkContext(conf)` with the real `spark.plugins` registration. It fails on the unfixed tree with that exception and passes with this change.
There are three places a boot-based test could live, each with a cost:
1. Direct call to `setPredefinedConfigs` (what this PR does). Same input state and same throwing line as the real path, runs in `gluten-core` with no extra setup. It just does not enter through `init`.
2. Full `SparkContext` boot in `gluten-core`. `GlutenDriverPlugin.init` calls `Component.sorted()`, which needs at least one `META-INF/gluten-components` file on the classpath, and `gluten-core` ships none (real backends live in the backend modules), so this needs a test-only dummy backend and a registration resource. `Component` also keeps a JVM-global graph with a one-shot load latch and no reset, and `ComponentSuite` registers an intentional dependency cycle without cleaning it up, so a boot suite sharing the test JVM hits `UnsupportedOperationException: Cycle detected in the component graph`. Making it stable needs a testing-only `Component` reset hook, with `ShuffleManagerRegistry.clear()` guarded by `SparkTestUtil.isTesting` as the precedent.
3. Integration test in a backend module like `backends-velox`. That module ships a real `VeloxBackend` component file and has no graph-polluting suite, so a real `spark.plugins` boot works there without any of the option 2 machinery. The cost is that it goes through `VeloxListenerApi.onDriverStart`, so it needs the native build and only runs in CI, and the test would sit in a different module than the one-line fix.
I lean toward option 1 for this fix, and option 3 if we want a production-level boot test. Fine either way, tell me which you prefer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12621][CORE] Fix NoSuchElementException in untracked memory mode without off-heap size [gluten]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #12622:
URL: https://github.com/apache/gluten/pull/12622#discussion_r3645768524
##########
gluten-core/src/main/scala/org/apache/gluten/GlutenPlugin.scala:
##########
@@ -134,6 +135,15 @@ private object GlutenDriverPlugin extends Logging {
((onHeapSize - (300 * 1024 * 1024)) *
conf.getDouble(GlutenCoreConfig.DYNAMIC_OFFHEAP_SIZING_MEMORY_FRACTION.key, 0.6d)).toLong
} else {
+ // Untracked memory mode skips the off-heap size requirement in checkOffHeapSettings, so
+ // the key may be absent here. Normalize it to 0 (mirroring the dynamic-sizing branch
+ // above) so downstream readers that read spark.memory.offHeap.size directly, e.g.
+ // VeloxListenerApi.onDriverStart, don't hit NoSuchElementException. Normal mode always has
+ // the key set because checkOffHeapSettings enforced it, so this only affects untracked
+ // mode without an explicit off-heap size.
+ if (!conf.contains(GlutenCoreConfig.SPARK_OFFHEAP_SIZE_KEY)) {
Review Comment:
we have checked the offHeap size while [initing GlutenPlugin](https://github.com/apache/gluten/blob/main/gluten-core/src/main/scala/org/apache/gluten/GlutenPlugin.scala#L96), this seems un neccesary? or we have any cases that meet this problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12621][CORE] Fix NoSuchElementException in untracked memory mode without off-heap size [gluten]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #12622:
URL: https://github.com/apache/gluten/pull/12622#issuecomment-5119638209
Thank you @jackylee-ch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org