You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "datakk2025 (via GitHub)" <gi...@apache.org> on 2025/08/06 02:24:51 UTC
[I] [Feature][API] 日志支持屏蔽数据库密码或者S3/OSS access-key和secert-key等敏感配置 [dolphinscheduler]
datakk2025 opened a new issue, #17398:
URL: https://github.com/apache/dolphinscheduler/issues/17398
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
_No response_
### Use case
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][API] Logs support blocking database passwords or sensitive configurations such as S3/OSS access-key and secert-key [dolphinscheduler]
Posted by "ylwf278 (via GitHub) via commits" <co...@dolphinscheduler.apache.org>.
ylwf278 commented on issue #17398:
URL: https://github.com/apache/dolphinscheduler/issues/17398#issuecomment-6913573171
Hi, I'd like to work on this issue.
Following up on @qiuyanjun888's analysis from June 22, I plan to go with Option B: extend `SensitiveDataConverter` with focused masking rules for S3/OSS keys (e.g. `access-key`, `secret-key`, `accessKey`, `secretKey`, `access.key.id`, `access.key.secret`), plus tests in `SensitiveDataConverterTest` covering JSON-like, `toString()`, and `key=value` formats.
I'll start with task instance logs only, and mask both access-key IDs and secret keys unless a maintainer says otherwise. If you prefer Option A (suppress the containing config log) instead, please let me know before I open the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][API] Logs support blocking database passwords or sensitive configurations such as S3/OSS access-key and secert-key [dolphinscheduler]
Posted by "qiuyanjun888 (via GitHub)" <gi...@apache.org>.
qiuyanjun888 commented on issue #17398:
URL: https://github.com/apache/dolphinscheduler/issues/17398#issuecomment-4766156217
I checked the existing issues/PRs and did not find another active issue or PR that directly duplicates this one.
I also noticed that there is already a task-log masking mechanism in `SensitiveDataConverter`, currently covering cases such as datasource `password`, DataX `post jdbc info`, Sqoop `--password`, and K8S `configYaml`. The maintainer comment also points to #17378, which suggests that we should first decide what should appear in task instance logs.
I think there are two possible implementation directions here:
## Option A: avoid logging sensitive configuration objects
If the leaked values come from printing a whole configuration object / initialization context, the safer fix may be to stop writing that object to the task instance log, or replace it with a smaller allowlisted summary. This follows the direction of #17378 and avoids relying only on masking after the sensitive text has already been formatted.
## Option B: extend the existing masking rules
If the log lines are legitimate and only specific key/value pairs need masking, then we can extend `SensitiveDataConverter` with focused patterns for common key names such as:
- `access.key.id`
- `access.key.secret`
- `access-key`
- `secret-key`
- `accessKey`
- `secretKey`
- possibly OSS/S3/COS-specific aliases if they appear in real logs
This should be covered by tests in `SensitiveDataConverterTest`, using the actual log formats we want to support, such as JSON-like strings, Java `toString()` style, and `key=value` / properties-like output.
## Questions before implementation
1. Which direction is preferred: suppress the containing configuration log, or keep the log and extend masking rules?
2. Could the reporter provide one real leaked Dinky task log line, with values redacted? I could not find a Dinky task plugin in the current repository, so a concrete sample would help create a minimal failing test.
3. Should access key IDs also be fully masked, or only secret keys? Some systems treat access key IDs as identifiers rather than secrets, but this issue mentions both access-key and secret-key.
4. Should this first fix be limited to task instance logs, or should it also cover remote logging / storage plugin logs?
If the preferred approach is Option B, I can start with focused unit tests in `SensitiveDataConverterTest` and then add the minimal masking rules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][API] Logs support blocking database passwords or sensitive configurations such as S3/OSS access-key and secert-key [dolphinscheduler]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #17398:
URL: https://github.com/apache/dolphinscheduler/issues/17398#issuecomment-3159946624
We can decide which log should appear in task instance log by using https://github.com/apache/dolphinscheduler/pull/17378
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][API] Logs support blocking database passwords or sensitive configurations such as S3/OSS access-key and secert-key [dolphinscheduler]
Posted by "datakk2025 (via GitHub)" <gi...@apache.org>.
datakk2025 commented on issue #17398:
URL: https://github.com/apache/dolphinscheduler/issues/17398#issuecomment-3157201046
在触发dinky任务时,日志中会出现数据库账号密码或者存储对象S3的access-key和secret-key等敏感配置
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org