You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dubbo.apache.org by "Qiao-yq (via GitHub)" <gi...@apache.org> on 2026/10/12 09:42:15 UTC
[PR] feat(ai): enforce sensitive data boundaries [dubbo-admin]
Qiao-yq opened a new pull request, #1594:
URL: https://github.com/apache/dubbo-admin/pull/1594
## Status
Draft. This PR is blocked by and must be merged after [#1592](https://github.com/apache/dubbo-admin/pull/1592), which establishes the Session ownership and access-isolation foundation used by this work.
## Summary
Implements the shared sensitive-data controls for Phase 1 of Issue #1539.
- add versioned data classification, sanitization, and safe logging primitives
- preserve canonical conversation messages while creating filtered model-input views
- keep normal logs and OpenTelemetry spans metadata-only
- enforce sensitive-content rejection and metadata allowlists for RAG indexes
- filter RAG query, embedding, and reranking provider inputs
- prevent provider, tool, stream, and storage errors from leaking through logs or HTTP/SSE responses
- document the security boundaries and deferred encryption/retention decisions
## Scope
This PR implements PR 2 of the Issue #1539 Phase 1 plan: unified security and sensitive-data controls.
Canonical conversation data remains unchanged. Filtering is applied only to ephemeral provider views, operational telemetry, and derived index boundaries. Baseline protections are always enabled and cannot be disabled by feature flags.
RAG indexing rejects an entire document when its content, external ID, or allowed metadata contains detected sensitive data. Query understanding, embeddings, and external reranking receive filtered temporary inputs, including when their lower-level APIs are called directly.
## Out of scope
- application-managed field encryption, key rotation, and ciphertext migration
- retention policy and deletion propagation, which belong to Phase 1 PR 3
- production identity adapters and resource-level authorization beyond #1592
- a privileged payload-capture or diagnostic channel
## Compatibility and behavior changes
- tracing `capture_content` now accepts only `none`; existing `truncated` or `full` configurations fail validation
- RAG indexing may reject documents that contain credentials or sensitive identifiers instead of silently modifying indexed evidence
- HTTP and SSE failures return stable public messages while detailed errors remain server-side as structured error types
- canonical Session messages are not rewritten or persisted in sanitized form
## Validation
- `cd ai && go test ./...`
- `cd ai && go vet ./...`
- `git diff --check`
- `gofmt -d` on changed Go files
- focused regression tests for model/RAG provider boundaries, sensitive document IDs, bounded encoded-secret detection, safe Finding paths, and HTTP/SSE error conversion
## Affected areas
- [x] Docs
- [ ] Installation
- [x] User Experience
- [ ] Dubboctl
- [x] Console
- [x] Core Component
## Characteristics
- [x] Includes tests
Part of #1539.
Blocked by #1592.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] feat(ai): enforce sensitive data boundaries [dubbo-admin]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #1594:
URL: https://github.com/apache/dubbo-admin/pull/1594#issuecomment-6121714013
## [](https://sonarcloud.io/dashboard?id=apache_dubbo-admin&pullRequest=1594) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_dubbo-admin&pullRequest=1594&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_dubbo-admin&pullRequest=1594&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_dubbo-admin&pullRequest=1594&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-admin&pullRequest=1594&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-admin&pullRequest=1594&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_dubbo-admin&pullRequest=1594)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] feat(ai): enforce sensitive data boundaries [dubbo-admin]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #1594:
URL: https://github.com/apache/dubbo-admin/pull/1594#issuecomment-6315763987
## [](https://sonarcloud.io/dashboard?id=apache_dubbo-admin&pullRequest=1594) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_dubbo-admin&pullRequest=1594&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_dubbo-admin&pullRequest=1594&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_dubbo-admin&pullRequest=1594&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-admin&pullRequest=1594&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-admin&pullRequest=1594&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_dubbo-admin&pullRequest=1594)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] feat(ai): enforce sensitive data boundaries [dubbo-admin]
Posted by "larry-zy (via GitHub)" <gi...@apache.org>.
larry-zy commented on PR #1594:
URL: https://github.com/apache/dubbo-admin/pull/1594#issuecomment-6383981878
**[P1] Redact credentials in JSON text before sending or indexing it**
Reviewed head: `0f7c8a6ee72098e6c7deee0404b73fb6c35ddcda`.
The credential-assignment pattern in [`ai/security/sanitizer.go:53`](https://github.com/apache/dubbo-admin/blob/0f7c8a6ee72098e6c7deee0404b73fb6c35ddcda/ai/security/sanitizer.go#L53) expects whitespace and an assignment separator immediately after the sensitive field name. It does not allow the closing quote between a JSON key and its colon. Consequently, this ordinary JSON text passes `SanitizeText` unchanged, with zero findings:
```json
{"password":"review-fixture-secret","enabled":true}
```
This affects real data boundaries, rather than only a standalone sanitizer API. [`modelInputView`](https://github.com/apache/dubbo-admin/blob/0f7c8a6ee72098e6c7deee0404b73fb6c35ddcda/ai/component/agent/react/model_view.go#L46-L58) filters message text with:
```go
part.Text = sanitizer.SanitizeText(part.Text).Value
```
A user pasting JSON configuration or a JSON log record therefore sends the credential to the model unchanged. The same text also passes `NewKnowledgeIndexPolicy().Prepare`, and a string log attribute containing it remains unredacted through `NewRedactingHandler`. Base64 encoding this JSON does not help: the decoded inspection uses the same pattern and also reports no findings. Structured Go maps are handled separately, so the existing nested-map tests do not cover these text cases.
I reproduced all of the above at this head using synthetic credentials: the model-input copy retains the password, derived-index validation accepts it, and the JSON log output contains it. The canonical message remains unchanged, as intended. The same sanitizer regressions also reproduce at #1611's reviewed head `04fe99de`.
Please handle quoted/escaped JSON sensitive keys at the text boundary, or safely inspect JSON text through the structured sanitizer, without rewriting canonical history. Add regressions for plain and encoded JSON at the model-input, logging, and derived-index boundaries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] feat(ai): enforce sensitive data boundaries [dubbo-admin]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #1594:
URL: https://github.com/apache/dubbo-admin/pull/1594#issuecomment-6392842453
## [](https://sonarcloud.io/dashboard?id=apache_dubbo-admin&pullRequest=1594) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_dubbo-admin&pullRequest=1594&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_dubbo-admin&pullRequest=1594&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_dubbo-admin&pullRequest=1594&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-admin&pullRequest=1594&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-admin&pullRequest=1594&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_dubbo-admin&pullRequest=1594)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] feat(ai): enforce sensitive data boundaries [dubbo-admin]
Posted by "larry-zy (via GitHub)" <gi...@apache.org>.
larry-zy commented on code in PR #1594:
URL: https://github.com/apache/dubbo-admin/pull/1594#discussion_r4376649960
##########
ai/security/sanitizer.go:
##########
@@ -0,0 +1,365 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package security
+
+import (
+ "encoding/base64"
+ "encoding/json"
+ "fmt"
+ "net/http"
+ "net/url"
+ "regexp"
+ "sort"
+ "strings"
+)
+
+const (
+ SanitizerRuleVersion = "2026-10-10.v1"
+ redactedValue = "[REDACTED]"
+ maxStructuredBytes = 1 << 20
+ maxStructuredDepth = 32
+)
+
+const (
+ RuleSensitiveField = "secret.sensitive-field"
+ RuleAuthorization = "secret.authorization"
+ RuleCredentialAssignment = "secret.credential-assignment"
+ RulePrivateKey = "secret.private-key"
+ RuleJWT = "secret.jwt"
+ RuleAPIKey = "secret.api-key"
+ RuleEncodedSecret = "secret.encoded"
+)
+
+var (
+ privateKeyPattern = regexp.MustCompile(`(?is)-----BEGIN [A-Z0-9 ]*PRIVATE KEY-----.*?-----END [A-Z0-9 ]*PRIVATE KEY-----`)
+ authorizationPattern = regexp.MustCompile(`(?i)\b(?:bearer|basic)\s+[A-Za-z0-9._~+/=-]+`)
+ jwtPattern = regexp.MustCompile(`\beyJ[A-Za-z0-9_-]{4,}\.[A-Za-z0-9_-]{4,}\.[A-Za-z0-9_-]{4,}\b`)
+ apiKeyPattern = regexp.MustCompile(`\b(?:sk-[A-Za-z0-9_-]{16,}|gh[pousr]_[A-Za-z0-9]{16,}|AIza[A-Za-z0-9_-]{20,}|xox[baprs]-[A-Za-z0-9-]{16,})\b`)
+ credentialAssignmentPattern = regexp.MustCompile(`(?i)\b(authorization|proxy[-_]?authorization|cookie|set[-_]?cookie|token|access[-_]?token|refresh[-_]?token|api[-_]?key|apikey|password|passwd|secret|client[-_]?secret|client[-_]?key[-_]?data|client[-_]?certificate[-_]?data)\b(\s*"?\s*(?::|=|%3[dD])\s*)(?:"[^"\r\n]*"|'[^'\r\n]*'|[^\s,;&\r\n]+)`)
Review Comment:
[P1] Consume escaped quotes as part of the credential value
The quoted-value alternative stops at an escaped quote inside a valid JSON string. For example, `{"password":"prefix\"review-suffix-secret","enabled":true}` is transformed into `{"password":"[REDACTED]"review-suffix-secret","enabled":true}`: the credential suffix is still visible. I reproduced this at this head through both `SanitizeText` and `modelInputView`, which forwards the surviving suffix to the model while leaving canonical history unchanged. Please make quoted-value consumption escape-aware, or parse valid JSON into a temporary structured view before redaction, and add escaped-quote/backslash cases to the model-boundary regression tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org