You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "parthchandra (via GitHub)" <gi...@apache.org> on 2025/06/02 16:23:57 UTC
[I] Improve integration of hadoop s3a and comet [datafusion-comet]
parthchandra opened a new issue, #1829:
URL: https://github.com/apache/datafusion-comet/issues/1829
### What is the problem the feature request solves?
https://github.com/apache/datafusion-comet/pull/1817 introduces integration that allows usage of `hadoop-aws`'s `s3a` configuration with the `object_store` used by the native scan implementations.
Some more work can be done to improve this -
1. The feature works with `native_datafusion` but not with `native_iceberg_compat`. To be truly useful we need the same integration in `native_iceberg_compat`.
2. To allow usage of custom authentication, we need a JNI based adapter that can get credentials from an AWS credentials provider (Both AWS SDK V1 and V2 credentials providers must be supported).
3. Document usage.
### Describe the potential solution
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve integration of hadoop s3a and comet [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on issue #1829:
URL: https://github.com/apache/datafusion-comet/issues/1829#issuecomment-5976314262
All three parts look done on main: `native_iceberg_compat` was removed (#4019, #4363), S3A credential providers work through the JVM bridge and the built-in adapters from #4309 and #6023, and `s3-credential-providers.md` documents it. I think this can be closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve integration of hadoop s3a and comet [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on issue #1829:
URL: https://github.com/apache/datafusion-comet/issues/1829#issuecomment-4393210497
That depends on whether we want to keep the `hdfs` implementation or whether we want to use `OpenDAL` exclusively. I'm not sure if anyone has spent time ensuring there are no gaps in coverage between the two implementations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve integration of hadoop s3a and comet [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1829:
URL: https://github.com/apache/datafusion-comet/issues/1829#issuecomment-4392149207
@parthchandra is this issue still valid?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org