[I] Native Delta scan declines Hadoop's S3A ProfileAWSCredentialsProvider [datafusion-comet]

["dwsmith1983 (via GitHub)" <gi...@apache.org>]

dwsmith1983 opened a new issue, #6931:
URL: https://github.com/apache/datafusion-comet/issues/6931

   ### What is the problem the feature request solves?
   
   #5872 lets native scans read S3 through Hadoop's `org.apache.hadoop.fs.s3a.auth.ProfileAWSCredentialsProvider`. When a bucket's `fs.s3a.aws.credentials.provider` list names it and `fs.s3a.comet.credential.provider.class` is unset, `s3.rs` hands the bucket to the built-in `HadoopS3ACredentialProviderAdapter`, which resolves the profile with Hadoop's own provider list.
   
   The native Delta scan checks provider class names at planning time, against [`SupportedCredentialProviderClasses`](https://github.com/apache/datafusion-comet/blob/474b5ece841a7fc6dff37ae04bd6787e654d7be7/contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala#L1663-L1681) in `DeltaScanSupport`, a copy of the names `s3.rs` accepts. The profile provider is not in that list, so once #5872 merges, a Delta table read with it still falls back to Spark. The results are correct; the table just doesn't get the native scan.
   
   ### Describe the potential solution
   
   Let the Delta gate accept `ProfileAWSCredentialsProvider` whenever `s3.rs` would route it to the adapter. The one case to keep declining is a bucket that opts out with a blank `fs.s3a.comet.credential.provider.class`: there `s3.rs` keeps its own reader and fails with `Unsupported credential provider` at execution time.
   
   The list's doc comment and [`nativeSupportedProviderClasses`](https://github.com/apache/datafusion-comet/blob/474b5ece841a7fc6dff37ae04bd6787e654d7be7/contrib/delta-spark/src/test/scala/org/apache/comet/contrib/delta/DeltaScanContribSuite.scala#L1562) in `DeltaScanContribSuite` would change with it, plus a test for the opt-out.
   
   ### Additional context
   
   Found in #5872 while merging main into it after the native Delta scan (#5365) landed. #5872 was written before the Delta contrib was on main and only changes the shared `s3.rs` path, so the Delta gate is left to this issue. It depends on #5872 merging first.
   
   ### Willingness to contribute
   
   I can contribute a fix for this bug independently
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org