You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "ddreyer (via GitHub)" <gi...@apache.org> on 2026/09/14 20:36:52 UTC
[PR] Redact credentials from captured audit payloads [pinot]
ddreyer opened a new pull request, #19555:
URL: https://github.com/apache/pinot/pull/19555
## Problem
`AuditRequestProcessor.captureRequestPayload` serializes the request entity, the query parameters and the allow-listed headers straight into the audit event. There is no redaction anywhere in `org.apache.pinot.common.audit`, so **no configuration can make payload capture safe**: any credential a caller sends to an audited endpoint is written verbatim to stdout and to the rotating audit file, then shipped wherever cluster logs go.
Observed on a controller with `capture.request.payload.enabled` set: every `POST /connections/browse` line carried the full body, exposing AWS access-key/secret-key pairs in plaintext next to the requesting principal. It is not specific to that endpoint — table creates carrying Kafka SASL passwords, warehouse passwords and tier-storage keys leak the same way. Anyone with log or node access can read live credentials, including log-pipeline operators who have no access to the API itself.
Capture is off by default (`capture.request.payload.enabled`, `AuditConfig`), but it is a supported feature that operators do turn on.
## Change
Adds `AuditRedactor`, through which everything captured from the caller now passes:
- **Credential-bearing keys are masked**, reusing `Obfuscator` with a pattern list extended past its suffix-anchored defaults. Those defaults miss the AWS `accessKey` / `accessKeyId` pair, which is exactly what leaked.
- **A body that is not parseable JSON is recorded by size only** (`[redacted: unparseable payload, N bytes]`). Secrets can only be located inside a structure that is understood, so an opaque payload is not kept at all. This includes bodies truncated at `request.payload.size.max.bytes`, whose trailing fragment is no longer valid JSON.
- **Redaction is unconditional** and has no configuration key, because every setting that disables it is a credential leak.
Key matching deliberately errs towards over-redaction: masking a harmless `tokenUri` costs an audit reader nothing, while missing one `accessKey` is the bug this prevents.
Query parameters and headers are redacted too — note that query parameters are captured regardless of `capture.request.payload.enabled`.
## Behavior change
Two existing tests asserted that an allow-listed `Authorization` / `X-Password` header was captured verbatim, while their own comments said it "should be filtered out". They now assert the mask. Allow-listing a credential-bearing header now records *that it was sent*, not what it said.
## Tests
- New `AuditRedactorTest`: data-driven coverage of sensitive vs. benign keys, nested objects, secrets inside arrays, top-level arrays, unparseable and truncated bodies, byte-vs-character sizing, and non-mutation of the input map.
- New cases in `AuditRequestProcessorTest` covering the end-to-end capture path, including that redaction does not disturb the entity stream handed to the resource method.
## Notes
This builds on the array-traversal fix already in `Obfuscator` on master — secrets nested inside JSON arrays (e.g. Kafka credentials in `streamConfigMaps`, a list of maps) depend on it.
The test suite was run green on an earlier base; it has not been re-run since rebasing onto current master. Marked draft pending that CI run.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Redact credentials from captured audit payloads [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #19555:
URL: https://github.com/apache/pinot/pull/19555#issuecomment-6220340543
Documentation follow-up merged: https://github.com/pinot-contrib/pinot-docs/pull/1115 — unconditional audit redaction, header allow-list behavior, structured/truncated payload handling, and residual embedded-secret risks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Redact credentials from captured audit payloads [pinot]
Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya merged PR #19555:
URL: https://github.com/apache/pinot/pull/19555
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Redact credentials from captured audit payloads [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19555:
URL: https://github.com/apache/pinot/pull/19555#issuecomment-6208806005
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/19555?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `87.77778%` with `22 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 58.89%. Comparing base ([`3101abc`](https://app.codecov.io/gh/apache/pinot/commit/3101abc90f74f279a6e77e4c1370e203fe7b4b16?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7d392c2`](https://app.codecov.io/gh/apache/pinot/commit/7d392c22fcd64fcfaed727c380c088b6d66ec200?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 3 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/19555?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...a/org/apache/pinot/common/audit/AuditRedactor.java](https://app.codecov.io/gh/apache/pinot/pull/19555?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Faudit%2FAuditRedactor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXVkaXQvQXVkaXRSZWRhY3Rvci5qYXZh) | 86.33% | [7 Missing and 15 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19555?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (3101abc) and HEAD (7d392c2). Click for more details.
>
> <details><summary>HEAD has 24 uploads less than BASE</summary>
>
>| Flag | BASE (3101abc) | HEAD (7d392c2) |
>|------|------|------|
>|unittests|2|1|
>|java-25|6|1|
>|temurin|6|1|
>|unittests2|1|0|
>|lane-a|2|0|
>|integration|4|0|
>|integration1|2|0|
>|lane-b|2|0|
>|integration2|2|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19555 +/- ##
=============================================
- Coverage 69.37% 58.89% -10.49%
+ Complexity 1491 1 -1490
=============================================
Files 3544 2728 -816
Lines 233553 171377 -62176
Branches 37353 28229 -9124
=============================================
- Hits 162030 100932 -61098
- Misses 59001 61937 +2936
+ Partials 12522 8508 -4014
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/19555/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [integration](https://app.codecov.io/gh/apache/pinot/pull/19555/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/19555/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/19555/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-25](https://app.codecov.io/gh/apache/pinot/pull/19555/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `58.89% <87.77%> (-10.49%)` | :arrow_down: |
| [lane-a](https://app.codecov.io/gh/apache/pinot/pull/19555/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [lane-b](https://app.codecov.io/gh/apache/pinot/pull/19555/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/19555/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `58.89% <87.77%> (-10.49%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/19555/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `58.89% <87.77%> (-10.48%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/19555/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `58.89% <87.77%> (+0.03%)` | :arrow_up: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/19555/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/19555?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org