User defined properties

[Omar Aloraini <al...@gmail.com>]

Hello all,
Apologies if this is an inappropriate place to ask questions, I tried to
use Slack but could sign in(I am either Slack illiterate, or I need some
sort of invite).

Thank you all for your work on this great project.

Following the page Spec page in the Table metadata fields section:

*optional* *optional* *properties* A string to string map of table
properties. This is used to control settings that affect reading and
writing and is not intended to be used for arbitrary metadata. For example,
commit.retry.num-retries is used to control the number of commit retries.

I followed the code path in PropertiesUpdate.commit, it seems that it's not
enforced that the table properties only contain iceberg defined properties,
the Spec page specifies that it should not be used for user metadata, are
their plans for relaxing such requirement as long no name collisions
happens, or introduce a 'user-properties' for user defined properties. My
use case is committing the source cursor along with the data files to get a
sort of 'exactly once' inserts.

Regards.

Re: User defined properties

[Russell Spitzer <ru...@gmail.com>]

Invite Link to ASF Slack

https://join.slack.com/t/the-asf/shared_invite/zt-ei391t4y-47FbOeNfu8SVA8BmSrtm8A

I believe the warning there is just to know it may have collisions in the
future or may be locked down, but I think it's pretty safe to actually add
your own properties there imho

On Mon, Nov 16, 2020 at 11:18 AM Omar Aloraini <al...@gmail.com>
wrote:

> Hello all,
> Apologies if this is an inappropriate place to ask questions, I tried to
> use Slack but could sign in(I am either Slack illiterate, or I need some
> sort of invite).
>
> Thank you all for your work on this great project.
>
> Following the page Spec page in the Table metadata fields section:
>
> *optional* *optional* *properties* A string to string map of table
> properties. This is used to control settings that affect reading and
> writing and is not intended to be used for arbitrary metadata. For example,
> commit.retry.num-retries is used to control the number of commit retries.
>
> I followed the code path in PropertiesUpdate.commit, it seems that it's
> not enforced that the table properties only contain iceberg defined
> properties, the Spec page specifies that it should not be used for user
> metadata, are their plans for relaxing such requirement as long no name
> collisions happens, or introduce a 'user-properties' for user defined
> properties. My use case is committing the source cursor along with the data
> files to get a sort of 'exactly once' inserts.
>
> Regards.
>