You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Xuanwo <xu...@apache.org> on 2024/07/10 07:19:16 UTC

[DISCUSS] Formalized File IO Properties

Hello everyone

I've been working on the iceberg-rust FileIO recently and have found it challenging to identify all the necessary IO properties we need to support.

For instance, consider AWS S3. There are no documents specifying which properties are supported by S3.

The only relevant documentation I could find includes:

- Iceberg AWS Integrations[1]: Does not define `s3.access-key-id` or `s3.secret-access-key`.
- Pyiceberg configuration[2]: Missing several S3-related properties.
- Iceberg REST Catalog[3]: Does not cover all storage services.

To gather this information, we must refer to the S3FileIO Java code[4].

I propose adding a separate section for agreeing upon these properties. We could create a specification that outlines all IO properties with indications of whether they are required or optional, along with their expected behaviors. This would help ensure consistency across different implementations without any conflicts.


[1]: https://iceberg.apache.org/docs/latest/aws/
[2]: https://py.iceberg.apache.org/configuration/#s3
[3]: https://github.com/apache/iceberg/blob/eee81c59199a54e749ea58dae070eb066d9a5f9e/open-api/rest-catalog-open-api.yaml#L2737
[4]: https://github.com/apache/iceberg/blob/2b21020aedb63c26295005d150c05f0a5a5f0eb2/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java#L46

Xuanwo

https://xuanwo.io/

Re: [DISCUSS] Formalized File IO Properties

Posted by Huang-Hsiang Cheng <hu...@apple.com.INVALID>.
Hi everyone,

I tried to consolidate S3 properties in this PR: https://github.com/apache/iceberg/pull/11321 

Hopefully we can start building a single source of truth from it.

Thanks for your review,
-Hsiang

> On Aug 7, 2024, at 3:44 AM, Kevin Liu <ke...@gmail.com> wrote:
> 
> +1 on standardizing, and possibly extending this to include catalog properties. 
> 
> On the PyIceberg side, a recent development is the ability to separate S3 FileIO configurations from the Glue Catalog configurations, with an optional configuration to use the same for both if specified. See Unified AWS Credentials <https://py.iceberg.apache.org/configuration/#unified-aws-credentials> and Github Issue #892 <https://github.com/apache/iceberg-python/issues/892>
> 
> So for AWS credentials, there are currently 3 different properties for `access-key-id`
> * `s3.access-key-id` (S3 FileIO specific) 
> * `glue.access-key-id` (Glue Catalog specific)
> * `client.access-key-id` (Unified)
> 
> Thanks,
> Kevin Liu
> 
> 
> 
> 
> On Wed, Jul 31, 2024 at 10:05 AM Xuanwo <xuanwo@apache.org <ma...@apache.org>> wrote:
>> Thanks you all. I'm going to prepare a proposal PR for this. 
>> 
>> On Fri, Jul 12, 2024, at 10:06, Honah J. wrote:
>>> Hello everyone,
>>> 
>>> Thank you all for the valuable insights. I am also +1 on having standardized names for File IO properties. Creating a dedicated section to summarize property names in the Java implementation is a good starting point. Since pyiceberg, icebergRust, and IcebergGolang will support only subsets of these properties for some time (with the rest to be added in future development), the existing Java implementation will serve as a useful reference. Additionally, we could establish general naming conventions in the doc, such as using the “s3.” prefix for S3 properties and hyphens to connect words.
>>> 
>>> Best regards,
>>> Honah
>>> 
>>> On Wed, Jul 10, 2024 at 10:47 AM <nd...@proton.me.invalid> wrote:
>>> 
>>> 
>>> I don't know what the recommended way to start standardizing is. We can start a proposal for each context or have one proposal to handle all.
>>> 
>>> Suggested contexts to start with:
>>> Rest Catalog
>>> FileIO
>>> 
>>> I believe that most of the other cases are supported by the configuration topic in the Table section[1], but this is about the Java implementation. Maybe we need to create a page in the project section[2] to handle the properties in the table section and the Rest and FileIO contexts.
>>> 
>>> 
>>> [1]: https://iceberg.apache.org/docs/latest/configuration/
>>> [2]: https://iceberg.apache.org/community/
>>> On Wednesday, July 10th, 2024 at 11:58 AM, Russell Spitzer <russell.spitzer@gmail.com <ma...@gmail.com>> wrote:
>>>> Sounds reasonable to me
>>>> 
>>>> On Wed, Jul 10, 2024 at 9:28 AM Renjie Liu <liurenjie2008@gmail.com <ma...@gmail.com>> wrote:
>>>> Hi:
>>>> 
>>>> +1 for standardizing iceberg properties. This will help to align different language implementations.
>>>> 
>>>> On Wed, Jul 10, 2024 at 9:44 PM <nd...@proton.me.invalid> wrote:
>>>> 
>>>> Hello Everyone,
>>>> 
>>>> I was considering discussing the standardization of Iceberg properties, and I believe this thread could be a great place to start.
>>>> 
>>>> I'm writing an Iceberg client in Elixir and using the Java, Python, and Rust implementations as references. However, I've had some difficulty determining which configurations we must support and what each client has implemented. Therefore, I agree with Xuanwo about having a separate section as a single source of truth (SSOT).
>>>> 
>>>> Additionally, I think it would be beneficial for each client to show what it does not support. This would make it easier for users to know that a particular client might not work with some configuration that their catalog could define as default or override. It would also help us, as contributors, to know which configurations we need to implement support for.
>>>> 
>>>> For example, the "s3.signer"[1] and "s3.proxy-uri"[2] configurations only exist in the Python implementation. I believe it is not clear that these configurations are exclusive to Python, and they might be configurations that the catalog could override or define as defaults in the get info endpoint. Without an SSOT, this could be harder to track.
>>>> 
>>>> Another example is the "rest.authorization-url" in Python and Rust versus "oauth2_server_uri" in Java. Although this is a bit out of scope for this thread, I will open another discussion topic about broader standardization of available properties.
>>>> 
>>>> 
>>>> [1]: https://github.com/search?q=repo%3Aapache%2Ficeberg-python+s3.signer&type=code
>>>> [2]: https://github.com/search?q=repo%3Aapache%2Ficeberg-python%20S3_PROXY_URI&type=code
>>>> 
>>>> On Wednesday, July 10th, 2024 at 7:51 AM, Fokko Driesprong <fokko@apache.org <ma...@apache.org>> wrote:
>>>>> Hey Xuanwo,
>>>>> 
>>>>> Thanks for raising this.
>>>>> The S3 properties are largely covered under the S3FileIO page: https://iceberg.apache.org/docs/nightly/aws/#s3-fileio. But it looks like some important ones are missing indeed. I've raised an issue here <https://github.com/apache/iceberg/issues/10674>.
>>>>> For PyIceberg it only supports like a subset of the functionality, and therefore also many properties are missing there.
>>>>> For the REST Catalog, there is an open PR to add <https://github.com/apache/iceberg/pull/10576> the options for GCS and ADLS. It would be great to get some more eyes on there.
>>>>> That being said, I do think there is value in formalizing them. When adding configuration options to PyIceberg, I'll make sure to check out the Java implementation to ensure that we use the same property.
>>>>> 
>>>>> Kind regards,
>>>>> Fokko
>>>>> 
>>>>> Op wo 10 jul 2024 om 09:22 schreef Xuanwo <xuanwo@apache.org <ma...@apache.org>>:
>>>>> Hello everyone
>>>>> 
>>>>> I've been working on the iceberg-rust FileIO recently and have found it challenging to identify all the necessary IO properties we need to support.
>>>>> 
>>>>> For instance, consider AWS S3. There are no documents specifying which properties are supported by S3.
>>>>> 
>>>>> The only relevant documentation I could find includes:
>>>>> 
>>>>> - Iceberg AWS Integrations[1]: Does not define `s3.access-key-id` or `s3.secret-access-key`.
>>>>> - Pyiceberg configuration[2]: Missing several S3-related properties.
>>>>> - Iceberg REST Catalog[3]: Does not cover all storage services.
>>>>> 
>>>>> To gather this information, we must refer to the S3FileIO Java code[4].
>>>>> 
>>>>> I propose adding a separate section for agreeing upon these properties. We could create a specification that outlines all IO properties with indications of whether they are required or optional, along with their expected behaviors. This would help ensure consistency across different implementations without any conflicts.
>>>>> 
>>>>> 
>>>>> [1]: https://iceberg.apache.org/docs/latest/aws/
>>>>> [2]: https://py.iceberg.apache.org/configuration/#s3
>>>>> [3]: https://github.com/apache/iceberg/blob/eee81c59199a54e749ea58dae070eb066d9a5f9e/open-api/rest-catalog-open-api.yaml#L2737
>>>>> [4]: https://github.com/apache/iceberg/blob/2b21020aedb63c26295005d150c05f0a5a5f0eb2/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java#L46
>>>>> 
>>>>> Xuanwo
>>>>> 
>>>>> https://xuanwo.io/
>> Xuanwo
>> 
>> https://xuanwo.io/
>> 


Re: [DISCUSS] Formalized File IO Properties

Posted by Kevin Liu <ke...@gmail.com>.
+1 on standardizing, and possibly extending this to include catalog
properties.

On the PyIceberg side, a recent development is the ability to separate S3
FileIO configurations from the Glue Catalog configurations, with an
optional configuration to use the same for both if specified. See Unified
AWS Credentials
<https://py.iceberg.apache.org/configuration/#unified-aws-credentials> and
Github Issue #892 <https://github.com/apache/iceberg-python/issues/892>

So for AWS credentials, there are currently 3 different properties for
`access-key-id`
* `s3.access-key-id` (S3 FileIO specific)
* `glue.access-key-id` (Glue Catalog specific)
* `client.access-key-id` (Unified)

Thanks,
Kevin Liu




On Wed, Jul 31, 2024 at 10:05 AM Xuanwo <xu...@apache.org> wrote:

> Thanks you all. I'm going to prepare a proposal PR for this.
>
> On Fri, Jul 12, 2024, at 10:06, Honah J. wrote:
>
> Hello everyone,
>
> Thank you all for the valuable insights. I am also +1 on having
> standardized names for File IO properties. Creating a dedicated section to
> summarize property names in the Java implementation is a good starting
> point. Since pyiceberg, icebergRust, and IcebergGolang will support only
> subsets of these properties for some time (with the rest to be added in
> future development), the existing Java implementation will serve as a
> useful reference. Additionally, we could establish general naming
> conventions in the doc, such as using the “s3.” prefix for S3 properties
> and hyphens to connect words.
>
> Best regards,
> Honah
>
> On Wed, Jul 10, 2024 at 10:47 AM <nd...@proton.me.invalid> wrote:
>
>
>
> I don't know what the recommended way to start standardizing is. We can
> start a proposal for each context or have one proposal to handle all.
>
> Suggested contexts to start with:
>
>    - Rest Catalog
>
>
>    - FileIO
>
>
> I believe that most of the other cases are supported by the configuration
> topic in the Table section[1], but this is about the Java implementation.
> Maybe we need to create a page in the project section[2] to handle the
> properties in the table section and the Rest and FileIO contexts.
>
>
> [1]: https://iceberg.apache.org/docs/latest/configuration/
> [2]: https://iceberg.apache.org/community/
> On Wednesday, July 10th, 2024 at 11:58 AM, Russell Spitzer <
> russell.spitzer@gmail.com> wrote:
>
> Sounds reasonable to me
>
> On Wed, Jul 10, 2024 at 9:28 AM Renjie Liu <li...@gmail.com>
> wrote:
>
> Hi:
>
> +1 for standardizing iceberg properties. This will help to align different
> language implementations.
>
> On Wed, Jul 10, 2024 at 9:44 PM <nd...@proton.me.invalid> wrote:
>
>
> Hello Everyone,
>
> I was considering discussing the standardization of Iceberg properties,
> and I believe this thread could be a great place to start.
>
> I'm writing an Iceberg client in Elixir and using the Java, Python, and
> Rust implementations as references. However, I've had some difficulty
> determining which configurations we must support and what each client has
> implemented. Therefore, I agree with Xuanwo about having a separate
> section as a single source of truth (SSOT).
>
> Additionally, I think it would be beneficial for each client to show what
> it does not support. This would make it easier for users to know that a
> particular client might not work with some configuration that their catalog
> could define as default or override. It would also help us, as
> contributors, to know which configurations we need to implement support for.
>
> For example, the "s3.signer"[1] and "s3.proxy-uri"[2] configurations only
> exist in the Python implementation. I believe it is not clear that these
> configurations are exclusive to Python, and they might be configurations
> that the catalog could override or define as defaults in the get info
> endpoint. Without an SSOT, this could be harder to track.
>
> Another example is the "rest.authorization-url" in Python and Rust versus
> "oauth2_server_uri" in Java. Although this is a bit out of scope for this
> thread, I will open another discussion topic about broader standardization
> of available properties.
>
>
> [1]:
> https://github.com/search?q=repo%3Aapache%2Ficeberg-python+s3.signer&type=code
> [2]:
> https://github.com/search?q=repo%3Aapache%2Ficeberg-python%20S3_PROXY_URI&type=code
>
> On Wednesday, July 10th, 2024 at 7:51 AM, Fokko Driesprong <
> fokko@apache.org> wrote:
>
> Hey Xuanwo,
>
> Thanks for raising this.
>
>    - The S3 properties are largely covered under the S3FileIO page:
>    https://iceberg.apache.org/docs/nightly/aws/#s3-fileio. But it looks
>    like some important ones are missing indeed. I've raised an issue here
>    <https://github.com/apache/iceberg/issues/10674>.
>    - For PyIceberg it only supports like a subset of the functionality,
>    and therefore also many properties are missing there.
>    - For the REST Catalog, there is an open PR to add
>    <https://github.com/apache/iceberg/pull/10576> the options for GCS and
>    ADLS. It would be great to get some more eyes on there.
>
> That being said, I do think there is value in formalizing them. When
> adding configuration options to PyIceberg, I'll make sure to check out the
> Java implementation to ensure that we use the same property.
>
> Kind regards,
> Fokko
>
> Op wo 10 jul 2024 om 09:22 schreef Xuanwo <xu...@apache.org>:
>
> Hello everyone
>
> I've been working on the iceberg-rust FileIO recently and have found it
> challenging to identify all the necessary IO properties we need to support.
>
> For instance, consider AWS S3. There are no documents specifying which
> properties are supported by S3.
>
> The only relevant documentation I could find includes:
>
> - Iceberg AWS Integrations[1]: Does not define `s3.access-key-id` or
> `s3.secret-access-key`.
> - Pyiceberg configuration[2]: Missing several S3-related properties.
> - Iceberg REST Catalog[3]: Does not cover all storage services.
>
> To gather this information, we must refer to the S3FileIO Java code[4].
>
> I propose adding a separate section for agreeing upon these properties. We
> could create a specification that outlines all IO properties with
> indications of whether they are required or optional, along with their
> expected behaviors. This would help ensure consistency across different
> implementations without any conflicts.
>
>
> [1]: https://iceberg.apache.org/docs/latest/aws/
> [2]: https://py.iceberg.apache.org/configuration/#s3
> [3]:
> https://github.com/apache/iceberg/blob/eee81c59199a54e749ea58dae070eb066d9a5f9e/open-api/rest-catalog-open-api.yaml#L2737
> [4]:
> https://github.com/apache/iceberg/blob/2b21020aedb63c26295005d150c05f0a5a5f0eb2/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java#L46
>
> Xuanwo
>
> https://xuanwo.io/
>
> Xuanwo
>
> https://xuanwo.io/
>
>

Re: [DISCUSS] Formalized File IO Properties

Posted by Xuanwo <xu...@apache.org>.
Thanks you all. I'm going to prepare a proposal PR for this. 

On Fri, Jul 12, 2024, at 10:06, Honah J. wrote:
> Hello everyone,
> 
> Thank you all for the valuable insights. I am also +1 on having standardized names for File IO properties. Creating a dedicated section to summarize property names in the Java implementation is a good starting point. Since pyiceberg, icebergRust, and IcebergGolang will support only subsets of these properties for some time (with the rest to be added in future development), the existing Java implementation will serve as a useful reference. Additionally, we could establish general naming conventions in the doc, such as using the “s3.” prefix for S3 properties and hyphens to connect words.
> 
> Best regards,
> Honah
> 
> On Wed, Jul 10, 2024 at 10:47 AM <nd...@proton.me.invalid> wrote:
>> 
>> 
>> I don't know what the recommended way to start standardizing is. We can start a proposal for each context or have one proposal to handle all.
>> 
>> Suggested contexts to start with:
>>  • Rest Catalog
>>  • FileIO
>> 
>> I believe that most of the other cases are supported by the configuration topic in the Table section[1], but this is about the Java implementation. Maybe we need to create a page in the project section[2] to handle the properties in the table section and the Rest and FileIO contexts.
>> 
>> 
>> [1]: https://iceberg.apache.org/docs/latest/configuration/
>> [2]: https://iceberg.apache.org/community/
>> On Wednesday, July 10th, 2024 at 11:58 AM, Russell Spitzer <ru...@gmail.com> wrote:
>>> Sounds reasonable to me
>>> 
>>> On Wed, Jul 10, 2024 at 9:28 AM Renjie Liu <li...@gmail.com> wrote:
>>>> Hi:
>>>> 
>>>> +1 for standardizing iceberg properties. This will help to align different language implementations.
>>>> 
>>>> On Wed, Jul 10, 2024 at 9:44 PM <nd...@proton.me.invalid> wrote:
>>>>> 
>>>>> Hello Everyone,
>>>>> 
>>>>> I was considering discussing the standardization of Iceberg properties, and I believe this thread could be a great place to start.
>>>>> 
>>>>> I'm writing an Iceberg client in Elixir and using the Java, Python, and Rust implementations as references. However, I've had some difficulty determining which configurations we must support and what each client has implemented. Therefore, I agree with Xuanwo about having a separate section as a single source of truth (SSOT).
>>>>> 
>>>>> Additionally, I think it would be beneficial for each client to show what it does not support. This would make it easier for users to know that a particular client might not work with some configuration that their catalog could define as default or override. It would also help us, as contributors, to know which configurations we need to implement support for.
>>>>> 
>>>>> For example, the "s3.signer"[1] and "s3.proxy-uri"[2] configurations only exist in the Python implementation. I believe it is not clear that these configurations are exclusive to Python, and they might be configurations that the catalog could override or define as defaults in the get info endpoint. Without an SSOT, this could be harder to track.
>>>>> 
>>>>> Another example is the "rest.authorization-url" in Python and Rust versus "oauth2_server_uri" in Java. Although this is a bit out of scope for this thread, I will open another discussion topic about broader standardization of available properties.
>>>>> 
>>>>> 
>>>>> [1]: https://github.com/search?q=repo%3Aapache%2Ficeberg-python+s3.signer&type=code
>>>>> [2]: https://github.com/search?q=repo%3Aapache%2Ficeberg-python%20S3_PROXY_URI&type=code
>>>>> 
>>>>> On Wednesday, July 10th, 2024 at 7:51 AM, Fokko Driesprong <fo...@apache.org> wrote:
>>>>>> Hey Xuanwo,
>>>>>> 
>>>>>> Thanks for raising this.
>>>>>>  • The S3 properties are largely covered under the S3FileIO page: https://iceberg.apache.org/docs/nightly/aws/#s3-fileio. But it looks like some important ones are missing indeed. I've raised an issue here <https://github.com/apache/iceberg/issues/10674>.
>>>>>>  • For PyIceberg it only supports like a subset of the functionality, and therefore also many properties are missing there.
>>>>>>  • For the REST Catalog, there is an open PR to add <https://github.com/apache/iceberg/pull/10576> the options for GCS and ADLS. It would be great to get some more eyes on there.
>>>>>> That being said, I do think there is value in formalizing them. When adding configuration options to PyIceberg, I'll make sure to check out the Java implementation to ensure that we use the same property.
>>>>>> 
>>>>>> Kind regards,
>>>>>> Fokko
>>>>>> 
>>>>>> Op wo 10 jul 2024 om 09:22 schreef Xuanwo <xu...@apache.org>:
>>>>>>> Hello everyone
>>>>>>> 
>>>>>>> I've been working on the iceberg-rust FileIO recently and have found it challenging to identify all the necessary IO properties we need to support.
>>>>>>> 
>>>>>>> For instance, consider AWS S3. There are no documents specifying which properties are supported by S3.
>>>>>>> 
>>>>>>> The only relevant documentation I could find includes:
>>>>>>> 
>>>>>>> - Iceberg AWS Integrations[1]: Does not define `s3.access-key-id` or `s3.secret-access-key`.
>>>>>>> - Pyiceberg configuration[2]: Missing several S3-related properties.
>>>>>>> - Iceberg REST Catalog[3]: Does not cover all storage services.
>>>>>>> 
>>>>>>> To gather this information, we must refer to the S3FileIO Java code[4].
>>>>>>> 
>>>>>>> I propose adding a separate section for agreeing upon these properties. We could create a specification that outlines all IO properties with indications of whether they are required or optional, along with their expected behaviors. This would help ensure consistency across different implementations without any conflicts.
>>>>>>> 
>>>>>>> 
>>>>>>> [1]: https://iceberg.apache.org/docs/latest/aws/
>>>>>>> [2]: https://py.iceberg.apache.org/configuration/#s3
>>>>>>> [3]: https://github.com/apache/iceberg/blob/eee81c59199a54e749ea58dae070eb066d9a5f9e/open-api/rest-catalog-open-api.yaml#L2737
>>>>>>> [4]: https://github.com/apache/iceberg/blob/2b21020aedb63c26295005d150c05f0a5a5f0eb2/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java#L46
>>>>>>> 
>>>>>>> Xuanwo
>>>>>>> 
>>>>>>> https://xuanwo.io/
Xuanwo

https://xuanwo.io/

Re: [DISCUSS] Formalized File IO Properties

Posted by "Honah J." <ho...@apache.org>.
Hello everyone,

Thank you all for the valuable insights. I am also +1 on having
standardized names for File IO properties. Creating a dedicated section to
summarize property names in the Java implementation is a good starting
point. Since pyiceberg, icebergRust, and IcebergGolang will support only
subsets of these properties for some time (with the rest to be added in
future development), the existing Java implementation will serve as a
useful reference. Additionally, we could establish general naming
conventions in the doc, such as using the “s3.” prefix for S3 properties
and hyphens to connect words.

Best regards,
Honah

On Wed, Jul 10, 2024 at 10:47 AM <nd...@proton.me.invalid> wrote:

> I don't know what the recommended way to start standardizing is. We can
> start a proposal for each context or have one proposal to handle all.
>
> Suggested contexts to start with:
>
>    - Rest Catalog
>
>
>    - FileIO
>
>
> I believe that most of the other cases are supported by the configuration
> topic in the Table section[1], but this is about the Java implementation.
> Maybe we need to create a page in the project section[2] to handle the
> properties in the table section and the Rest and FileIO contexts.
>
> [1]: https://iceberg.apache.org/docs/latest/configuration/
> [2]: https://iceberg.apache.org/community/
> On Wednesday, July 10th, 2024 at 11:58 AM, Russell Spitzer <
> russell.spitzer@gmail.com> wrote:
>
> Sounds reasonable to me
>
> On Wed, Jul 10, 2024 at 9:28 AM Renjie Liu <li...@gmail.com>
> wrote:
>
>> Hi:
>>
>> +1 for standardizing iceberg properties. This will help to align
>> different language implementations.
>>
>> On Wed, Jul 10, 2024 at 9:44 PM <nd...@proton.me.invalid> wrote:
>>
>>> Hello Everyone,
>>>
>>> I was considering discussing the standardization of Iceberg properties,
>>> and I believe this thread could be a great place to start.
>>>
>>> I'm writing an Iceberg client in Elixir and using the Java, Python, and
>>> Rust implementations as references. However, I've had some difficulty
>>> determining which configurations we must support and what each client has
>>> implemented. Therefore, I agree with Xuanwo about having a separate
>>> section as a single source of truth (SSOT).
>>>
>>> Additionally, I think it would be beneficial for each client to show
>>> what it does not support. This would make it easier for users to know that
>>> a particular client might not work with some configuration that their
>>> catalog could define as default or override. It would also help us, as
>>> contributors, to know which configurations we need to implement support for.
>>>
>>> For example, the "s3.signer"[1] and "s3.proxy-uri"[2] configurations
>>> only exist in the Python implementation. I believe it is not clear that
>>> these configurations are exclusive to Python, and they might be
>>> configurations that the catalog could override or define as defaults in the
>>> get info endpoint. Without an SSOT, this could be harder to track.
>>>
>>> Another example is the "rest.authorization-url" in Python and Rust
>>> versus "oauth2_server_uri" in Java. Although this is a bit out of scope for
>>> this thread, I will open another discussion topic about broader
>>> standardization of available properties.
>>>
>>> [1]:
>>> https://github.com/search?q=repo%3Aapache%2Ficeberg-python+s3.signer&type=code
>>> [2]:
>>> https://github.com/search?q=repo%3Aapache%2Ficeberg-python%20S3_PROXY_URI&type=code
>>> On Wednesday, July 10th, 2024 at 7:51 AM, Fokko Driesprong <
>>> fokko@apache.org> wrote:
>>>
>>> Hey Xuanwo,
>>>
>>> Thanks for raising this.
>>>
>>>    - The S3 properties are largely covered under the S3FileIO page:
>>>    https://iceberg.apache.org/docs/nightly/aws/#s3-fileio. But it looks
>>>    like some important ones are missing indeed. I've raised an issue
>>>    here <https://github.com/apache/iceberg/issues/10674>.
>>>    - For PyIceberg it only supports like a subset of the functionality,
>>>    and therefore also many properties are missing there.
>>>    - For the REST Catalog, there is an open PR to add
>>>    <https://github.com/apache/iceberg/pull/10576> the options for GCS
>>>    and ADLS. It would be great to get some more eyes on there.
>>>
>>> That being said, I do think there is value in formalizing them. When
>>> adding configuration options to PyIceberg, I'll make sure to check out the
>>> Java implementation to ensure that we use the same property.
>>>
>>> Kind regards,
>>> Fokko
>>>
>>> Op wo 10 jul 2024 om 09:22 schreef Xuanwo <xu...@apache.org>:
>>>
>>>> Hello everyone
>>>>
>>>> I've been working on the iceberg-rust FileIO recently and have found it
>>>> challenging to identify all the necessary IO properties we need to support.
>>>>
>>>> For instance, consider AWS S3. There are no documents specifying which
>>>> properties are supported by S3.
>>>>
>>>> The only relevant documentation I could find includes:
>>>>
>>>> - Iceberg AWS Integrations[1]: Does not define `s3.access-key-id` or
>>>> `s3.secret-access-key`.
>>>> - Pyiceberg configuration[2]: Missing several S3-related properties.
>>>> - Iceberg REST Catalog[3]: Does not cover all storage services.
>>>>
>>>> To gather this information, we must refer to the S3FileIO Java code[4].
>>>>
>>>> I propose adding a separate section for agreeing upon these properties.
>>>> We could create a specification that outlines all IO properties with
>>>> indications of whether they are required or optional, along with their
>>>> expected behaviors. This would help ensure consistency across different
>>>> implementations without any conflicts.
>>>>
>>>>
>>>> [1]: https://iceberg.apache.org/docs/latest/aws/
>>>> [2]: https://py.iceberg.apache.org/configuration/#s3
>>>> [3]:
>>>> https://github.com/apache/iceberg/blob/eee81c59199a54e749ea58dae070eb066d9a5f9e/open-api/rest-catalog-open-api.yaml#L2737
>>>> [4]:
>>>> https://github.com/apache/iceberg/blob/2b21020aedb63c26295005d150c05f0a5a5f0eb2/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java#L46
>>>>
>>>> Xuanwo
>>>>
>>>> https://xuanwo.io/
>>>>
>>>
>>>
>

Re: [DISCUSS] Formalized File IO Properties

Posted by nd...@proton.me.INVALID.
I don't know what the recommended way to start standardizing is. We can start a proposal for each context or have one proposal to handle all.

Suggested contexts to start with:

- Rest Catalog

- FileIO

I believe that most of the other cases are supported by the configuration topic in the Table section[1], but this is about the Java implementation. Maybe we need to create a page in the project section[2] to handle the properties in the table section and the Rest and FileIO contexts.

[1]: https://iceberg.apache.org/docs/latest/configuration/
[2]: https://iceberg.apache.org/community/
On Wednesday, July 10th, 2024 at 11:58 AM, Russell Spitzer <ru...@gmail.com> wrote:

> Sounds reasonable to me
>
> On Wed, Jul 10, 2024 at 9:28 AM Renjie Liu <li...@gmail.com> wrote:
>
>> Hi:
>>
>> +1 for standardizing iceberg properties. This will help to align different language implementations.
>>
>> On Wed, Jul 10, 2024 at 9:44 PM <nd...@proton.me.invalid> wrote:
>>
>>> Hello Everyone,
>>>
>>> I was considering discussing the standardization of Iceberg properties, and I believe this thread could be a great place to start.
>>>
>>> I'm writing an Iceberg client in Elixir and using the Java, Python, and Rust implementations as references. However, I've had some difficulty determining which configurations we must support and what each client has implemented. Therefore, I agree with Xuanwo about having a separate section as a single source of truth (SSOT).
>>>
>>> Additionally, I think it would be beneficial for each client to show what it does not support. This would make it easier for users to know that a particular client might not work with some configuration that their catalog could define as default or override. It would also help us, as contributors, to know which configurations we need to implement support for.
>>>
>>> For example, the "s3.signer"[1] and "s3.proxy-uri"[2] configurations only exist in the Python implementation. I believe it is not clear that these configurations are exclusive to Python, and they might be configurations that the catalog could override or define as defaults in the get info endpoint. Without an SSOT, this could be harder to track.
>>> Another example is the "rest.authorization-url" in Python and Rust versus "oauth2_server_uri" in Java. Although this is a bit out of scope for this thread, I will open another discussion topic about broader standardization of available properties.
>>>
>>> [1]: https://github.com/search?q=repo%3Aapache%2Ficeberg-python+s3.signer&type=code
>>> [2]: https://github.com/search?q=repo%3Aapache%2Ficeberg-python%20S3_PROXY_URI&type=code
>>>
>>> On Wednesday, July 10th, 2024 at 7:51 AM, Fokko Driesprong <fo...@apache.org> wrote:
>>>
>>>> Hey Xuanwo,
>>>>
>>>> Thanks for raising this.
>>>>
>>>> - The S3 properties are largely covered under the S3FileIO page: https://iceberg.apache.org/docs/nightly/aws/#s3-fileio. But it looks like some important ones are missing indeed. I've raised [an issue here](https://github.com/apache/iceberg/issues/10674).
>>>> - For PyIceberg it only supports like a subset of the functionality, and therefore also many properties are missing there.
>>>> - For the REST Catalog, there is [an open PR to add](https://github.com/apache/iceberg/pull/10576) the options for GCS and ADLS. It would be great to get some more eyes on there.
>>>>
>>>> That being said, I do think there is value in formalizing them. When adding configuration options to PyIceberg, I'll make sure to check out the Java implementation to ensure that we use the same property.
>>>>
>>>> Kind regards,
>>>> Fokko
>>>>
>>>> Op wo 10 jul 2024 om 09:22 schreef Xuanwo <xu...@apache.org>:
>>>>
>>>>> Hello everyone
>>>>>
>>>>> I've been working on the iceberg-rust FileIO recently and have found it challenging to identify all the necessary IO properties we need to support.
>>>>>
>>>>> For instance, consider AWS S3. There are no documents specifying which properties are supported by S3.
>>>>>
>>>>> The only relevant documentation I could find includes:
>>>>>
>>>>> - Iceberg AWS Integrations[1]: Does not define `s3.access-key-id` or `s3.secret-access-key`.
>>>>> - Pyiceberg configuration[2]: Missing several S3-related properties.
>>>>> - Iceberg REST Catalog[3]: Does not cover all storage services.
>>>>>
>>>>> To gather this information, we must refer to the S3FileIO Java code[4].
>>>>>
>>>>> I propose adding a separate section for agreeing upon these properties. We could create a specification that outlines all IO properties with indications of whether they are required or optional, along with their expected behaviors. This would help ensure consistency across different implementations without any conflicts.
>>>>>
>>>>> [1]: https://iceberg.apache.org/docs/latest/aws/
>>>>> [2]: https://py.iceberg.apache.org/configuration/#s3
>>>>> [3]: https://github.com/apache/iceberg/blob/eee81c59199a54e749ea58dae070eb066d9a5f9e/open-api/rest-catalog-open-api.yaml#L2737
>>>>> [4]: https://github.com/apache/iceberg/blob/2b21020aedb63c26295005d150c05f0a5a5f0eb2/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java#L46
>>>>>
>>>>> Xuanwo
>>>>>
>>>>> https://xuanwo.io/

Re: [DISCUSS] Formalized File IO Properties

Posted by Alex Dutra <al...@dremio.com.INVALID>.
Hi,

Also +1 on standardizing properties. I'm looking forward to this discussion
topic. In particular, REST catalog properties imho should be standardized
with the "rest." prefix, and REST auth properties should imho have a prefix
like "rest.auth.<auth-type>.", e.g. "rest.auth.oauth2.issuer-url".

Thanks,
Alex

On Wed, Jul 10, 2024 at 4:59 PM Russell Spitzer <ru...@gmail.com>
wrote:

> Sounds reasonable to me
>
> On Wed, Jul 10, 2024 at 9:28 AM Renjie Liu <li...@gmail.com>
> wrote:
>
>> Hi:
>>
>> +1 for standardizing iceberg properties. This will help to align
>> different language implementations.
>>
>> On Wed, Jul 10, 2024 at 9:44 PM <nd...@proton.me.invalid> wrote:
>>
>>> Hello Everyone,
>>>
>>> I was considering discussing the standardization of Iceberg properties,
>>> and I believe this thread could be a great place to start.
>>>
>>> I'm writing an Iceberg client in Elixir and using the Java, Python, and
>>> Rust implementations as references. However, I've had some difficulty
>>> determining which configurations we must support and what each client has
>>> implemented. Therefore, I agree with Xuanwo about having a separate
>>> section as a single source of truth (SSOT).
>>>
>>> Additionally, I think it would be beneficial for each client to show
>>> what it does not support. This would make it easier for users to know that
>>> a particular client might not work with some configuration that their
>>> catalog could define as default or override. It would also help us, as
>>> contributors, to know which configurations we need to implement support for.
>>>
>>> For example, the "s3.signer"[1] and "s3.proxy-uri"[2] configurations
>>> only exist in the Python implementation. I believe it is not clear that
>>> these configurations are exclusive to Python, and they might be
>>> configurations that the catalog could override or define as defaults in the
>>> get info endpoint. Without an SSOT, this could be harder to track.
>>>
>>> Another example is the "rest.authorization-url" in Python and Rust
>>> versus "oauth2_server_uri" in Java. Although this is a bit out of scope for
>>> this thread, I will open another discussion topic about broader
>>> standardization of available properties.
>>>
>>> [1]:
>>> https://github.com/search?q=repo%3Aapache%2Ficeberg-python+s3.signer&type=code
>>> [2]:
>>> https://github.com/search?q=repo%3Aapache%2Ficeberg-python%20S3_PROXY_URI&type=code
>>> On Wednesday, July 10th, 2024 at 7:51 AM, Fokko Driesprong <
>>> fokko@apache.org> wrote:
>>>
>>> Hey Xuanwo,
>>>
>>> Thanks for raising this.
>>>
>>>    - The S3 properties are largely covered under the S3FileIO page:
>>>    https://iceberg.apache.org/docs/nightly/aws/#s3-fileio. But it looks
>>>    like some important ones are missing indeed. I've raised an issue
>>>    here <https://github.com/apache/iceberg/issues/10674>.
>>>    - For PyIceberg it only supports like a subset of the functionality,
>>>    and therefore also many properties are missing there.
>>>    - For the REST Catalog, there is an open PR to add
>>>    <https://github.com/apache/iceberg/pull/10576> the options for GCS
>>>    and ADLS. It would be great to get some more eyes on there.
>>>
>>> That being said, I do think there is value in formalizing them. When
>>> adding configuration options to PyIceberg, I'll make sure to check out the
>>> Java implementation to ensure that we use the same property.
>>>
>>> Kind regards,
>>> Fokko
>>>
>>> Op wo 10 jul 2024 om 09:22 schreef Xuanwo <xu...@apache.org>:
>>>
>>>> Hello everyone
>>>>
>>>> I've been working on the iceberg-rust FileIO recently and have found it
>>>> challenging to identify all the necessary IO properties we need to support.
>>>>
>>>> For instance, consider AWS S3. There are no documents specifying which
>>>> properties are supported by S3.
>>>>
>>>> The only relevant documentation I could find includes:
>>>>
>>>> - Iceberg AWS Integrations[1]: Does not define `s3.access-key-id` or
>>>> `s3.secret-access-key`.
>>>> - Pyiceberg configuration[2]: Missing several S3-related properties.
>>>> - Iceberg REST Catalog[3]: Does not cover all storage services.
>>>>
>>>> To gather this information, we must refer to the S3FileIO Java code[4].
>>>>
>>>> I propose adding a separate section for agreeing upon these properties.
>>>> We could create a specification that outlines all IO properties with
>>>> indications of whether they are required or optional, along with their
>>>> expected behaviors. This would help ensure consistency across different
>>>> implementations without any conflicts.
>>>>
>>>>
>>>> [1]: https://iceberg.apache.org/docs/latest/aws/
>>>> [2]: https://py.iceberg.apache.org/configuration/#s3
>>>> [3]:
>>>> https://github.com/apache/iceberg/blob/eee81c59199a54e749ea58dae070eb066d9a5f9e/open-api/rest-catalog-open-api.yaml#L2737
>>>> [4]:
>>>> https://github.com/apache/iceberg/blob/2b21020aedb63c26295005d150c05f0a5a5f0eb2/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java#L46
>>>>
>>>> Xuanwo
>>>>
>>>> https://xuanwo.io/
>>>>
>>>
>>>

Re: [DISCUSS] Formalized File IO Properties

Posted by Russell Spitzer <ru...@gmail.com>.
Sounds reasonable to me

On Wed, Jul 10, 2024 at 9:28 AM Renjie Liu <li...@gmail.com> wrote:

> Hi:
>
> +1 for standardizing iceberg properties. This will help to align different
> language implementations.
>
> On Wed, Jul 10, 2024 at 9:44 PM <nd...@proton.me.invalid> wrote:
>
>> Hello Everyone,
>>
>> I was considering discussing the standardization of Iceberg properties,
>> and I believe this thread could be a great place to start.
>>
>> I'm writing an Iceberg client in Elixir and using the Java, Python, and
>> Rust implementations as references. However, I've had some difficulty
>> determining which configurations we must support and what each client has
>> implemented. Therefore, I agree with Xuanwo about having a separate
>> section as a single source of truth (SSOT).
>>
>> Additionally, I think it would be beneficial for each client to show what
>> it does not support. This would make it easier for users to know that a
>> particular client might not work with some configuration that their catalog
>> could define as default or override. It would also help us, as
>> contributors, to know which configurations we need to implement support for.
>>
>> For example, the "s3.signer"[1] and "s3.proxy-uri"[2] configurations only
>> exist in the Python implementation. I believe it is not clear that these
>> configurations are exclusive to Python, and they might be configurations
>> that the catalog could override or define as defaults in the get info
>> endpoint. Without an SSOT, this could be harder to track.
>>
>> Another example is the "rest.authorization-url" in Python and Rust versus
>> "oauth2_server_uri" in Java. Although this is a bit out of scope for this
>> thread, I will open another discussion topic about broader standardization
>> of available properties.
>>
>> [1]:
>> https://github.com/search?q=repo%3Aapache%2Ficeberg-python+s3.signer&type=code
>> [2]:
>> https://github.com/search?q=repo%3Aapache%2Ficeberg-python%20S3_PROXY_URI&type=code
>> On Wednesday, July 10th, 2024 at 7:51 AM, Fokko Driesprong <
>> fokko@apache.org> wrote:
>>
>> Hey Xuanwo,
>>
>> Thanks for raising this.
>>
>>    - The S3 properties are largely covered under the S3FileIO page:
>>    https://iceberg.apache.org/docs/nightly/aws/#s3-fileio. But it looks
>>    like some important ones are missing indeed. I've raised an issue here
>>    <https://github.com/apache/iceberg/issues/10674>.
>>    - For PyIceberg it only supports like a subset of the functionality,
>>    and therefore also many properties are missing there.
>>    - For the REST Catalog, there is an open PR to add
>>    <https://github.com/apache/iceberg/pull/10576> the options for GCS
>>    and ADLS. It would be great to get some more eyes on there.
>>
>> That being said, I do think there is value in formalizing them. When
>> adding configuration options to PyIceberg, I'll make sure to check out the
>> Java implementation to ensure that we use the same property.
>>
>> Kind regards,
>> Fokko
>>
>> Op wo 10 jul 2024 om 09:22 schreef Xuanwo <xu...@apache.org>:
>>
>>> Hello everyone
>>>
>>> I've been working on the iceberg-rust FileIO recently and have found it
>>> challenging to identify all the necessary IO properties we need to support.
>>>
>>> For instance, consider AWS S3. There are no documents specifying which
>>> properties are supported by S3.
>>>
>>> The only relevant documentation I could find includes:
>>>
>>> - Iceberg AWS Integrations[1]: Does not define `s3.access-key-id` or
>>> `s3.secret-access-key`.
>>> - Pyiceberg configuration[2]: Missing several S3-related properties.
>>> - Iceberg REST Catalog[3]: Does not cover all storage services.
>>>
>>> To gather this information, we must refer to the S3FileIO Java code[4].
>>>
>>> I propose adding a separate section for agreeing upon these properties.
>>> We could create a specification that outlines all IO properties with
>>> indications of whether they are required or optional, along with their
>>> expected behaviors. This would help ensure consistency across different
>>> implementations without any conflicts.
>>>
>>>
>>> [1]: https://iceberg.apache.org/docs/latest/aws/
>>> [2]: https://py.iceberg.apache.org/configuration/#s3
>>> [3]:
>>> https://github.com/apache/iceberg/blob/eee81c59199a54e749ea58dae070eb066d9a5f9e/open-api/rest-catalog-open-api.yaml#L2737
>>> [4]:
>>> https://github.com/apache/iceberg/blob/2b21020aedb63c26295005d150c05f0a5a5f0eb2/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java#L46
>>>
>>> Xuanwo
>>>
>>> https://xuanwo.io/
>>>
>>
>>

Re: [DISCUSS] Formalized File IO Properties

Posted by Renjie Liu <li...@gmail.com>.
Hi:

+1 for standardizing iceberg properties. This will help to align different
language implementations.

On Wed, Jul 10, 2024 at 9:44 PM <nd...@proton.me.invalid> wrote:

> Hello Everyone,
>
> I was considering discussing the standardization of Iceberg properties,
> and I believe this thread could be a great place to start.
>
> I'm writing an Iceberg client in Elixir and using the Java, Python, and
> Rust implementations as references. However, I've had some difficulty
> determining which configurations we must support and what each client has
> implemented. Therefore, I agree with Xuanwo about having a separate
> section as a single source of truth (SSOT).
>
> Additionally, I think it would be beneficial for each client to show what
> it does not support. This would make it easier for users to know that a
> particular client might not work with some configuration that their catalog
> could define as default or override. It would also help us, as
> contributors, to know which configurations we need to implement support for.
>
> For example, the "s3.signer"[1] and "s3.proxy-uri"[2] configurations only
> exist in the Python implementation. I believe it is not clear that these
> configurations are exclusive to Python, and they might be configurations
> that the catalog could override or define as defaults in the get info
> endpoint. Without an SSOT, this could be harder to track.
>
> Another example is the "rest.authorization-url" in Python and Rust versus
> "oauth2_server_uri" in Java. Although this is a bit out of scope for this
> thread, I will open another discussion topic about broader standardization
> of available properties.
>
> [1]:
> https://github.com/search?q=repo%3Aapache%2Ficeberg-python+s3.signer&type=code
> [2]:
> https://github.com/search?q=repo%3Aapache%2Ficeberg-python%20S3_PROXY_URI&type=code
> On Wednesday, July 10th, 2024 at 7:51 AM, Fokko Driesprong <
> fokko@apache.org> wrote:
>
> Hey Xuanwo,
>
> Thanks for raising this.
>
>    - The S3 properties are largely covered under the S3FileIO page:
>    https://iceberg.apache.org/docs/nightly/aws/#s3-fileio. But it looks
>    like some important ones are missing indeed. I've raised an issue here
>    <https://github.com/apache/iceberg/issues/10674>.
>    - For PyIceberg it only supports like a subset of the functionality,
>    and therefore also many properties are missing there.
>    - For the REST Catalog, there is an open PR to add
>    <https://github.com/apache/iceberg/pull/10576> the options for GCS and
>    ADLS. It would be great to get some more eyes on there.
>
> That being said, I do think there is value in formalizing them. When
> adding configuration options to PyIceberg, I'll make sure to check out the
> Java implementation to ensure that we use the same property.
>
> Kind regards,
> Fokko
>
> Op wo 10 jul 2024 om 09:22 schreef Xuanwo <xu...@apache.org>:
>
>> Hello everyone
>>
>> I've been working on the iceberg-rust FileIO recently and have found it
>> challenging to identify all the necessary IO properties we need to support.
>>
>> For instance, consider AWS S3. There are no documents specifying which
>> properties are supported by S3.
>>
>> The only relevant documentation I could find includes:
>>
>> - Iceberg AWS Integrations[1]: Does not define `s3.access-key-id` or
>> `s3.secret-access-key`.
>> - Pyiceberg configuration[2]: Missing several S3-related properties.
>> - Iceberg REST Catalog[3]: Does not cover all storage services.
>>
>> To gather this information, we must refer to the S3FileIO Java code[4].
>>
>> I propose adding a separate section for agreeing upon these properties.
>> We could create a specification that outlines all IO properties with
>> indications of whether they are required or optional, along with their
>> expected behaviors. This would help ensure consistency across different
>> implementations without any conflicts.
>>
>>
>> [1]: https://iceberg.apache.org/docs/latest/aws/
>> [2]: https://py.iceberg.apache.org/configuration/#s3
>> [3]:
>> https://github.com/apache/iceberg/blob/eee81c59199a54e749ea58dae070eb066d9a5f9e/open-api/rest-catalog-open-api.yaml#L2737
>> [4]:
>> https://github.com/apache/iceberg/blob/2b21020aedb63c26295005d150c05f0a5a5f0eb2/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java#L46
>>
>> Xuanwo
>>
>> https://xuanwo.io/
>>
>
>

Re: [DISCUSS] Formalized File IO Properties

Posted by nd...@proton.me.INVALID.
Hello Everyone,

I was considering discussing the standardization of Iceberg properties, and I believe this thread could be a great place to start.

I'm writing an Iceberg client in Elixir and using the Java, Python, and Rust implementations as references. However, I've had some difficulty determining which configurations we must support and what each client has implemented. Therefore, I agree with Xuanwo about having a separate section as a single source of truth (SSOT).

Additionally, I think it would be beneficial for each client to show what it does not support. This would make it easier for users to know that a particular client might not work with some configuration that their catalog could define as default or override. It would also help us, as contributors, to know which configurations we need to implement support for.

For example, the "s3.signer"[1] and "s3.proxy-uri"[2] configurations only exist in the Python implementation. I believe it is not clear that these configurations are exclusive to Python, and they might be configurations that the catalog could override or define as defaults in the get info endpoint. Without an SSOT, this could be harder to track.
Another example is the "rest.authorization-url" in Python and Rust versus "oauth2_server_uri" in Java. Although this is a bit out of scope for this thread, I will open another discussion topic about broader standardization of available properties.

[1]: https://github.com/search?q=repo%3Aapache%2Ficeberg-python+s3.signer&type=code
[2]: https://github.com/search?q=repo%3Aapache%2Ficeberg-python%20S3_PROXY_URI&type=code

On Wednesday, July 10th, 2024 at 7:51 AM, Fokko Driesprong <fo...@apache.org> wrote:

> Hey Xuanwo,
>
> Thanks for raising this.
>
> - The S3 properties are largely covered under the S3FileIO page: https://iceberg.apache.org/docs/nightly/aws/#s3-fileio. But it looks like some important ones are missing indeed. I've raised [an issue here](https://github.com/apache/iceberg/issues/10674).
> - For PyIceberg it only supports like a subset of the functionality, and therefore also many properties are missing there.
> - For the REST Catalog, there is [an open PR to add](https://github.com/apache/iceberg/pull/10576) the options for GCS and ADLS. It would be great to get some more eyes on there.
>
> That being said, I do think there is value in formalizing them. When adding configuration options to PyIceberg, I'll make sure to check out the Java implementation to ensure that we use the same property.
>
> Kind regards,
> Fokko
>
> Op wo 10 jul 2024 om 09:22 schreef Xuanwo <xu...@apache.org>:
>
>> Hello everyone
>>
>> I've been working on the iceberg-rust FileIO recently and have found it challenging to identify all the necessary IO properties we need to support.
>>
>> For instance, consider AWS S3. There are no documents specifying which properties are supported by S3.
>>
>> The only relevant documentation I could find includes:
>>
>> - Iceberg AWS Integrations[1]: Does not define `s3.access-key-id` or `s3.secret-access-key`.
>> - Pyiceberg configuration[2]: Missing several S3-related properties.
>> - Iceberg REST Catalog[3]: Does not cover all storage services.
>>
>> To gather this information, we must refer to the S3FileIO Java code[4].
>>
>> I propose adding a separate section for agreeing upon these properties. We could create a specification that outlines all IO properties with indications of whether they are required or optional, along with their expected behaviors. This would help ensure consistency across different implementations without any conflicts.
>>
>> [1]: https://iceberg.apache.org/docs/latest/aws/
>> [2]: https://py.iceberg.apache.org/configuration/#s3
>> [3]: https://github.com/apache/iceberg/blob/eee81c59199a54e749ea58dae070eb066d9a5f9e/open-api/rest-catalog-open-api.yaml#L2737
>> [4]: https://github.com/apache/iceberg/blob/2b21020aedb63c26295005d150c05f0a5a5f0eb2/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java#L46
>>
>> Xuanwo
>>
>> https://xuanwo.io/

Re: [DISCUSS] Formalized File IO Properties

Posted by Fokko Driesprong <fo...@apache.org>.
Hey Xuanwo,

Thanks for raising this.

   - The S3 properties are largely covered under the S3FileIO page:
   https://iceberg.apache.org/docs/nightly/aws/#s3-fileio. But it looks
   like some important ones are missing indeed. I've raised an issue here
   <https://github.com/apache/iceberg/issues/10674>.
   - For PyIceberg it only supports like a subset of the functionality, and
   therefore also many properties are missing there.
   - For the REST Catalog, there is an open PR to add
   <https://github.com/apache/iceberg/pull/10576> the options for GCS and
   ADLS. It would be great to get some more eyes on there.

That being said, I do think there is value in formalizing them. When adding
configuration options to PyIceberg, I'll make sure to check out the Java
implementation to ensure that we use the same property.

Kind regards,
Fokko

Op wo 10 jul 2024 om 09:22 schreef Xuanwo <xu...@apache.org>:

> Hello everyone
>
> I've been working on the iceberg-rust FileIO recently and have found it
> challenging to identify all the necessary IO properties we need to support.
>
> For instance, consider AWS S3. There are no documents specifying which
> properties are supported by S3.
>
> The only relevant documentation I could find includes:
>
> - Iceberg AWS Integrations[1]: Does not define `s3.access-key-id` or
> `s3.secret-access-key`.
> - Pyiceberg configuration[2]: Missing several S3-related properties.
> - Iceberg REST Catalog[3]: Does not cover all storage services.
>
> To gather this information, we must refer to the S3FileIO Java code[4].
>
> I propose adding a separate section for agreeing upon these properties. We
> could create a specification that outlines all IO properties with
> indications of whether they are required or optional, along with their
> expected behaviors. This would help ensure consistency across different
> implementations without any conflicts.
>
>
> [1]: https://iceberg.apache.org/docs/latest/aws/
> [2]: https://py.iceberg.apache.org/configuration/#s3
> [3]:
> https://github.com/apache/iceberg/blob/eee81c59199a54e749ea58dae070eb066d9a5f9e/open-api/rest-catalog-open-api.yaml#L2737
> [4]:
> https://github.com/apache/iceberg/blob/2b21020aedb63c26295005d150c05f0a5a5f0eb2/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java#L46
>
> Xuanwo
>
> https://xuanwo.io/
>