You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@livy.apache.org by "soumyadeeplogin (via GitHub)" <gi...@apache.org> on 2026/09/17 06:25:10 UTC
[PR] [LIVY-1082] Add config to disable Kubernetes driver pod log polling [livy]
soumyadeeplogin opened a new pull request, #557:
URL: https://github.com/apache/livy/pull/557
## What changes were proposed in this pull request?
`SparkKubernetesApp` currently fetches the driver pod's Kubernetes log (`pods/<driver>/log`) on every poll cycle to populate live log lines in `/sessions/:id/log` and `/batches/:id/log` responses. On large clusters, or when driver logs are already collected externally (e.g. via a centralized log pipeline / Spark History Server), this is wasted request volume against the Kubernetes API server, and can be a source of transient errors during pod lifecycle transitions.
This PR adds a new config, `livy.server.kubernetes.driver-log-polling.enabled` (default `true`, preserving existing behavior), that when set to `false` skips the `pods/<driver>/log` fetch while Livy continues to poll pod state and diagnostics normally.
## Why are the changes needed?
To let operators opt out of the driver log poll when it's unnecessary, reducing Kubernetes API server load and avoiding transient log-fetch errors during pod lifecycle transitions, without losing pod state/diagnostics monitoring.
## Does this PR introduce _any_ user-facing change?
Yes. A new optional config `livy.server.kubernetes.driver-log-polling.enabled` is added (default `true`). When explicitly set to `false`, `/sessions/:id/log` and `/batches/:id/log` will no longer include live Kubernetes driver log lines; pod state and diagnostics continue to be polled and reported as before.
## How was this patch tested?
Added/updated unit tests in `SparkKubernetesAppSpec`:
- `should enable driver log polling by default`
- `resolveDriverAppLog` - `should skip fetching the log when driver log polling is disabled`
- `resolveDriverAppLog` - `should fetch the log when driver log polling is enabled`
Ran `mvn -pl server -am test -Dtest=SparkKubernetesAppSpec -DwildcardSuites=org.apache.livy.utils.SparkKubernetesAppSpec`; all 14 tests in the suite pass.
## Was this patch authored or co-authored using generative AI tooling?
Yes, this patch was co-authored using Claude Code (Anthropic).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1082] Add config to disable Kubernetes driver pod log polling [livy]
Posted by "soumyadeeplogin (via GitHub)" <gi...@apache.org>.
soumyadeeplogin commented on PR #557:
URL: https://github.com/apache/livy/pull/557#issuecomment-5709945411
Filed LIVY-1082 for this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1082] Add config to disable Kubernetes driver pod log polling [livy]
Posted by "gyogal (via GitHub)" <gi...@apache.org>.
gyogal commented on code in PR #557:
URL: https://github.com/apache/livy/pull/557#discussion_r4048352368
##########
conf/livy.conf.template:
##########
@@ -294,6 +294,10 @@
# per-executor diagnostics). Disable on large clusters; driver pod is polled regardless.
# livy.server.kubernetes.executor-tracking.enabled = true
+# Whether Livy fetches the driver pod's Kubernetes logs each poll cycle. Disable if driver
+# logs are collected externally and freshness in /sessions/:id/log, /batches/:id/log is not needed.
+# livy.server.kubernetes.driver-log-polling.enabled = true
Review Comment:
Your change looks good, I only have a minor observation here on the documentation: I think "freshness is not needed" and "won't include fresh Kubernetes driver logs." later on implies that there will some kind of logs, just not fresh. However if I understand correctly, based on the changes, there will not be any driver logs recorded in this case. During testing, did "/logs" return anything? If not, I think it would be better to indicate that logs will not be available at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1082] Add config to disable Kubernetes driver pod log polling [livy]
Posted by "gyogal (via GitHub)" <gi...@apache.org>.
gyogal merged PR #557:
URL: https://github.com/apache/livy/pull/557
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1082] Add config to disable Kubernetes driver pod log polling [livy]
Posted by "soumyadeeplogin (via GitHub)" <gi...@apache.org>.
soumyadeeplogin commented on code in PR #557:
URL: https://github.com/apache/livy/pull/557#discussion_r4048430087
##########
conf/livy.conf.template:
##########
@@ -294,6 +294,10 @@
# per-executor diagnostics). Disable on large clusters; driver pod is polled regardless.
# livy.server.kubernetes.executor-tracking.enabled = true
+# Whether Livy fetches the driver pod's Kubernetes logs each poll cycle. Disable if driver
+# logs are collected externally and freshness in /sessions/:id/log, /batches/:id/log is not needed.
+# livy.server.kubernetes.driver-log-polling.enabled = true
Review Comment:
You're right, thanks for catching that. Confirmed: when driver-log-polling is disabled, resolveDriverAppLog always returns an empty log, so kubernetesAppLog is permanently empty and /sessions/:id/log and /batches/:id/log return no driver log lines at all (not stale ones). Updated the doc wording in both livy.conf.template and LivyConf.scala to say so explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1082] Add config to disable Kubernetes driver pod log polling [livy]
Posted by "soumyadeeplogin (via GitHub)" <gi...@apache.org>.
soumyadeeplogin commented on PR #557:
URL: https://github.com/apache/livy/pull/557#issuecomment-5726412667
Hi @gyogal / @roczei — could one of you approve the pending CI workflow run for this PR (fork-contributor approval gate)? Happy to address any review feedback as well. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1082] Add config to disable Kubernetes driver pod log polling [livy]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #557:
URL: https://github.com/apache/livy/pull/557#issuecomment-5733978515
## [Codecov](https://app.codecov.io/gh/apache/livy/pull/557?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `22.22222%` with `7 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 53.28%. Comparing base ([`501e822`](https://app.codecov.io/gh/apache/livy/commit/501e8226cbb36d6ae00d00bb208e778c9328ff72?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`fe8e84c`](https://app.codecov.io/gh/apache/livy/commit/fe8e84cb57f062ea9091a7b0e8fccd12e4b45aee?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 12 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/livy/pull/557?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ala/org/apache/livy/utils/SparkKubernetesApp.scala](https://app.codecov.io/gh/apache/livy/pull/557?src=pr&el=tree&filepath=server%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Flivy%2Futils%2FSparkKubernetesApp.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS91dGlscy9TcGFya0t1YmVybmV0ZXNBcHAuc2NhbGE=) | 0.00% | [7 Missing :warning: ](https://app.codecov.io/gh/apache/livy/pull/557?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #557 +/- ##
=============================================
- Coverage 68.68% 53.28% -15.41%
+ Complexity 1218 845 -373
=============================================
Files 106 106
Lines 6815 6816 +1
Branches 836 845 +9
=============================================
- Hits 4681 3632 -1049
- Misses 1666 2742 +1076
+ Partials 468 442 -26
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/livy/pull/557?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org