Re: [PR] fix: decline native scans whose S3 configuration diverges from Hadoop's [datafusion-comet]

["sunchao (via GitHub)" <gi...@apache.org>]

sunchao commented on code in PR #7026:
URL: https://github.com/apache/datafusion-comet/pull/7026#discussion_r4331657673


##########
spark/src/main/scala/org/apache/comet/objectstore/NativeConfig.scala:
##########
@@ -428,6 +444,12 @@ object NativeConfig {
           "Native scan does not support ABFS files from more than one container or account " +
             s"(${first.getRawAuthority} and ${uri.getRawAuthority})")
       }
+      collectS3(uri)
+    }
+    val s3Reason =
+      if (s3Uris.isEmpty) None else S3ConfigGate.declineReason(hadoopConf, s3Uris.values.toSeq)

Review Comment:
   [P2] Preserve supported SPI-backed scans when applying the gate here. Configure `fs.s3a.comet.credential.provider.class=org.apache.comet.cloud.s3.HadoopS3ACredentialProviderAdapter`, `fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider`, and credentials only under `fs.s3a.bucket.<bucket>.fs.s3a.access.key` and `.secret.key`. The real Comet adapter resolves these long-form aliases exactly as Hadoop does, so the existing native scan can use the same credentials as Spark. However, `s3KeyDivergenceReason` still compares against `nativeShortThenGlobal`, which cannot see those aliases, and this new invocation returns `Left`, forcing regular Parquet/CSV scans back to Spark. This disables native scanning for a previously supported configuration. Model the active SPI credential-resolution path in the equivalence check while retaining the store-level checks, and add an adapter-backed regression test.
   
   Evidence: An offline probe compiled the exact-head `S3ConfigGate`, `HadoopS3ACredentialProviderAdapter` and supporting classes, then compared adapter credentials with the real Hadoop `SimpleAWSCredentialsProvider` after `S3AUtils.propagateBucketOptions`. With synthetic global credentials, both providers matched and the gate returned `None`. With the same synthetic credentials stored only in long-form bucket aliases, both providers still matched, but the gate returned `Some(Native scan cannot forward fs.s3a.access.key for review-test-bucket ...)`. The probe exited successfully after asserting credential equality. Source: `/tmp/comet7026-review/AdapterGateProbe.scala`; output: `/tmp/comet7026-review/probe.log`. Native `s3.rs::create_store` selects the SPI bridge instead of `build_credential_provider` when this adapter is configured. The base version of `extractScanObjectStoreOptions` did not invoke the gate.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org