You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2025/06/22 17:53:17 UTC

[PR] feat: Mask sensitive task configuration values in logs [pinot]

xiangfu0 opened a new pull request, #16171:
URL: https://github.com/apache/pinot/pull/16171

   
   This change prevents sensitive credentials from being exposed in task execution logs while preserving all non-sensitive configuration information for debugging purposes.
   
   - Integrate existing Obfuscator utility to mask sensitive config values in TaskFactoryRegistry logs
   - Replace direct logging of pinotTaskConfig.getConfigs() with obfuscated JSON output
   - Add comprehensive test coverage for Obfuscator functionality
   - Enhance Obfuscator patterns to include 'api_key' variations
   - Ensure sensitive values like authToken, password, secretKey, apiKey are masked as '*****'
   - Maintain existing functionality while improving security in log output
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] feat: Mask sensitive task configuration values in logs [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #16171:
URL: https://github.com/apache/pinot/pull/16171#discussion_r2160825997


##########
pinot-minion/src/main/java/org/apache/pinot/minion/taskfactory/TaskFactoryRegistry.java:
##########
@@ -146,7 +147,7 @@ private TaskResult runInternal(PinotTaskConfig pinotTaskConfig) {
 
               _eventObserver.notifyTaskStart(pinotTaskConfig);
               LOGGER.info("Start running {}: {} with configs: {}", pinotTaskConfig.getTaskType(), _taskConfig.getId(),
-                  pinotTaskConfig.getConfigs());
+                  new Obfuscator().toJsonString(pinotTaskConfig.getConfigs()));

Review Comment:
   This will obfuscate the text even if the log is not logged, which would be a waste of resources.
   
   In SLF4J the only way to solve the issue is to do something like:
   ```java
   if (LOGGER.isInfoEnabled()) {
     LOGGER.info(whatever, usually using the " + value +" syntax instead of {})
   }
   ```
   
   If instead of SLF4J, which is very old, we were using log4j2 apis to log, we could use lambdas to log, like:
   ```
   LOGGER.trace("Start running {}: {} with configs: {}", pinotTaskConfig.getTaskType(), _taskConfig.getId(), () ->  new Obfuscator().toJsonString(pinotTaskConfig.getConfigs())));
   ```
   
   Given the log is at info level (which is usually enabled) and the fact that this is only called during startup, this is probably not actually problematic. So I don't think this inefficiency is a blocker, but in general I would recommend using the pattern above in cases like this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] feat: Mask sensitive task configuration values in logs [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #16171:
URL: https://github.com/apache/pinot/pull/16171#discussion_r2160845453


##########
pinot-minion/src/main/java/org/apache/pinot/minion/taskfactory/TaskFactoryRegistry.java:
##########
@@ -146,7 +147,7 @@ private TaskResult runInternal(PinotTaskConfig pinotTaskConfig) {
 
               _eventObserver.notifyTaskStart(pinotTaskConfig);
               LOGGER.info("Start running {}: {} with configs: {}", pinotTaskConfig.getTaskType(), _taskConfig.getId(),
-                  pinotTaskConfig.getConfigs());
+                  new Obfuscator().toJsonString(pinotTaskConfig.getConfigs()));

Review Comment:
   make sense
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] feat: Mask sensitive task configuration values in logs [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 merged PR #16171:
URL: https://github.com/apache/pinot/pull/16171


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] feat: Mask sensitive task configuration values in logs [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #16171:
URL: https://github.com/apache/pinot/pull/16171#issuecomment-2994374476

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/16171?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: Patch coverage is `50.00000%` with `1 line` in your changes missing coverage. Please review.
   > Project coverage is 33.32%. Comparing base [(`1a476de`)](https://app.codecov.io/gh/apache/pinot/commit/1a476dea72ebf1c15e14ded28ae60c385f71b2c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`0b9f744`)](https://app.codecov.io/gh/apache/pinot/commit/0b9f7440446b34463aa0fea4aa086fdad0d063da?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 311 commits behind head on master.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/16171?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [.../pinot/minion/taskfactory/TaskFactoryRegistry.java](https://app.codecov.io/gh/apache/pinot/pull/16171?src=pr&el=tree&filepath=pinot-minion%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fminion%2Ftaskfactory%2FTaskFactoryRegistry.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vdGFza2ZhY3RvcnkvVGFza0ZhY3RvcnlSZWdpc3RyeS5qYXZh) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/16171?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   > :exclamation:  There is a different number of reports uploaded between BASE (1a476de) and HEAD (0b9f744). Click for more details.
   > 
   > <details><summary>HEAD has 68 uploads less than BASE</summary>
   >
   >| Flag | BASE (1a476de) | HEAD (0b9f744) |
   >|------|------|------|
   >|java-21|10|1|
   >|unittests|6|1|
   >|unittests1|3|0|
   >|skip-bytebuffers-true|4|0|
   >|temurin|15|1|
   >|unittests2|3|1|
   >|skip-bytebuffers-false|8|0|
   >|java-11|5|0|
   >|integration|9|0|
   >|integration2|3|0|
   >|integration1|3|0|
   >|custom-integration1|3|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@              Coverage Diff              @@
   ##             master   #16171       +/-   ##
   =============================================
   - Coverage     62.90%   33.32%   -29.58%     
   + Complexity     1386      659      -727     
   =============================================
     Files          2867     2953       +86     
     Lines        163354   169916     +6562     
     Branches      24952    25985     +1033     
   =============================================
   - Hits         102755    56632    -46123     
   - Misses        52847   108738    +55891     
   + Partials       7752     4546     -3206     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/16171/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/16171/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/16171/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/16171/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/16171/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/16171/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/16171/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.32% <50.00%> (-29.50%)` | :arrow_down: |
   | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/16171/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/16171/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/16171/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.32% <50.00%> (-29.58%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/16171/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.32% <50.00%> (-29.58%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/16171/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/16171/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.32% <50.00%> (-0.25%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/16171?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org