You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "nhuantho (via GitHub)" <gi...@apache.org> on 2025/12/22 17:18:57 UTC
[PR] [doc] Add highlight note for Hadoop S3A FileSystem [iceberg]
nhuantho opened a new pull request, #14913:
URL: https://github.com/apache/iceberg/pull/14913
## Why:
* Docs of Iceberg mentioned about recommended using S3FileIO instead of HadoopFileIO, but it is not highlighted. So I want to add a highlight note for Hadoop S3A FileSystem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [doc] Add highlight note for Hadoop S3A FileSystem [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #14913:
URL: https://github.com/apache/iceberg/pull/14913#discussion_r2644885471
##########
docs/docs/aws.md:
##########
@@ -416,9 +416,11 @@ There is no redundant consistency wait and check which might negatively impact p
### Hadoop S3A FileSystem
+!!! important
+ **S3FileIO is recommended** for S3 use cases rather than the `S3A FileSystem` (`HadoopFileIO`).
Review Comment:
```suggestion
**S3FileIO is recommended** for S3 use cases rather than the `S3A FileSystem` (`HadoopFileIO`). `S3FileIO` adopts the latest AWS clients and S3 features for optimized security and performance.
```
nit: i prefer to move the sentence below to here since as a reader id like to know why its recommended :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [doc] Add highlight note for Hadoop S3A FileSystem [iceberg]
Posted by "nhuantho (via GitHub)" <gi...@apache.org>.
nhuantho commented on code in PR #14913:
URL: https://github.com/apache/iceberg/pull/14913#discussion_r2644940896
##########
docs/docs/aws.md:
##########
@@ -416,9 +416,11 @@ There is no redundant consistency wait and check which might negatively impact p
### Hadoop S3A FileSystem
+!!! important
+ **S3FileIO is recommended** for S3 use cases rather than the `S3A FileSystem` (`HadoopFileIO`).
Review Comment:
@kevinjqliu I only want to emphasize that: **S3FileIO is recommended** for S3 use cases rather than the `S3A FileSystem` (`HadoopFileIO`), which is the reason I used an important note. And people need to spend 1 minute to read the main reason, so I think that it is not necessary to add the important note.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [doc] Add highlight note for Hadoop S3A FileSystem [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #14913:
URL: https://github.com/apache/iceberg/pull/14913#discussion_r2646116154
##########
docs/docs/aws.md:
##########
@@ -416,9 +416,11 @@ There is no redundant consistency wait and check which might negatively impact p
### Hadoop S3A FileSystem
+!!! important
+ **S3FileIO is recommended** for S3 use cases rather than the `S3A FileSystem` (`HadoopFileIO`).
Review Comment:
makes sense, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [doc] Add highlight note for Hadoop S3A FileSystem [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu merged PR #14913:
URL: https://github.com/apache/iceberg/pull/14913
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [doc] Add highlight note for Hadoop S3A FileSystem [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #14913:
URL: https://github.com/apache/iceberg/pull/14913#discussion_r2646077569
##########
docs/docs/aws.md:
##########
@@ -416,9 +416,11 @@ There is no redundant consistency wait and check which might negatively impact p
### Hadoop S3A FileSystem
+!!! important
+ **S3FileIO is recommended** for S3 use cases rather than the `S3A FileSystem` (`HadoopFileIO`).
Review Comment:
My main reason would also be different than the one we had before, the HadoopFileIO does many "hadoopy posixy" things that are in many cases not what you want. It's best to work with the Object store apis directly. The path resolution issue that @nhuantho originally noted is one of those weirdnesses but there is also "empty directory markers" and a bunch of other things that I think we are constantly working around
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [doc] Add highlight note for Hadoop S3A FileSystem [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #14913:
URL: https://github.com/apache/iceberg/pull/14913#discussion_r2640994964
##########
docs/docs/aws.md:
##########
@@ -416,9 +416,13 @@ There is no redundant consistency wait and check which might negatively impact p
### Hadoop S3A FileSystem
+> ⚠️ **Important**
Review Comment:
This isn't how we do note boxes
See
https://github.com/apache/iceberg/blob/1c5bb017c2161aa568d75e17a4de7f42bcb37d02/docs/docs/spark-ddl.md#L352-L353
or
https://github.com/apache/iceberg/blob/1c5bb017c2161aa568d75e17a4de7f42bcb37d02/docs/docs/spark-ddl.md#L375-L377
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [doc] Add highlight note for Hadoop S3A FileSystem [iceberg]
Posted by "nhuantho (via GitHub)" <gi...@apache.org>.
nhuantho commented on PR #14913:
URL: https://github.com/apache/iceberg/pull/14913#issuecomment-3694881234
Close https://github.com/apache/iceberg/issues/14863
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [doc] Add highlight note for Hadoop S3A FileSystem [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #14913:
URL: https://github.com/apache/iceberg/pull/14913#issuecomment-3683834634
I have no problem with moving the text as I mentioned but we should probably use the established info boxes, also please be sure to actually render the doc page to show the changes you are making. Sometimes it's hard to catch things just from the Markdown
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [doc] Add highlight note for Hadoop S3A FileSystem [iceberg]
Posted by "nhuantho (via GitHub)" <gi...@apache.org>.
nhuantho commented on PR #14913:
URL: https://github.com/apache/iceberg/pull/14913#issuecomment-3684702951
> I have no problem with moving the text as I mentioned but we should probably use the established info boxes, also please be sure to actually render the doc page to show the changes you are making. Sometimes it's hard to catch things just from the Markdown
@RussellSpitzer I changed to info boxes, and added an image about actually render the doc page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org