[I] Access credentials provided in ingestion specs may becomes unavailable for pending tasks during overlord switch (druid)

["a2l007 (via GitHub)" <gi...@apache.org>]

a2l007 opened a new issue, #15357:
URL: https://github.com/apache/druid/issues/15357

   ### Affected Version
   
   Identified in Druid 26.0, possibly affects versions starting 24.0
   
   ### Description
   Currently there are three input sources: `s3`, `http` and `oss` which provide the flexibility to provide access credentials as part of the input source spec. For example:
   ```
   "inputSource": {
           "type": "s3",
           "prefixes": [
             "s3://uploadpath/1699551000"
           ],
           "properties": {
             "accessKeyId": {
               "type": "default",
               "password": <accesskey>
             },
             "secretAccessKey": {
               "type": "default",
               "password": <secret>
             }
           }
         },
   ```
   When such tasks are in pending state and an overlord leadership change happens, the new leader no longer has access to the `password` provided in the ingestion spec. Consequently, once these ingestion tasks start, they fail while attempting to connect to the input source.
   The reason for this behavior is that when the task spec is written to the metadata store, it [redacts the password from the spec](https://github.com/apache/druid/blob/master/server/src/main/java/org/apache/druid/metadata/SQLMetadataStorageActionHandler.java#L105) to avoid leaks. As a result, when the new leader deserializes the task object, the password is no longer there.
   
   A fix for this would be to match the mixin only for `DefaultPasswordProvider` instead of the `PasswordProvider`. 
   ```
       this.jsonMapper = jsonMapper.copy().addMixIn(org.apache.druid.metadata.DefaultPasswordProvider.class,
               org.apache.druid.metadata.PasswordProviderRedactionMixIn.class);
   ```
   This will still redact the password for the default provider, but will retain the password object in the task spec for other providers such as `environment`.
   I'm raising this issue to see if there are any better ways of fixing it and if there are any security implications that I may have overlooked with the above solution.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org