You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "advancedxy (via GitHub)" <gi...@apache.org> on 2023/08/08 15:01:14 UTC
[GitHub] [iceberg] advancedxy opened a new issue, #8258: multi-arg transform support
advancedxy opened a new issue, #8258:
URL: https://github.com/apache/iceberg/issues/8258
### Feature Request / Improvement
As discussed in #5626, it would be nice to have multi-arg transform supported in Iceberg, especially for bucket transform.
I wrote up a design doc for this improvement: https://docs.google.com/document/d/1aDoZqRgvDOOUVAGhvKZbp5vFstjsAMY4EFCyjlxpaaw/edit?usp=sharing
Quoted background from the doc:
> Iceberg uses a transform to produce partitioning value from a source value. Currently the supported transforms are: `Years`, `Months`, `Days`, `Hours`, `Identity`, `Void`, `Truncate`, `Bucket`. Since the current spec requires that each partitioning field consists of a source column id in the table’s schema, the above transforms only accept one argument as its input. However, it’s possible and quite common to use multiple arguments to produce a partitioning value, especially for the `Bucket` transform. Other transforms might require multiple arguments in the future. This document tries to add multi-arg transform support in Iceberg, especially for the bucket transform.
------
I also did a poc version of how multiple arg bucket would be supported in Spark: . Some places are not modified yet, such as UpdatePartitionSpec, TableMetadata related.
I'd like to get feedbacks from the community before going too much further.
If we have reached the consensus that we should support multi-arg transform and the spec changes are stabilized after reviewing. I would update my code accordingly, and extend the Flink engine support.
### Query engine
Spark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] advancedxy commented on issue #8258: multi-arg transform support
Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on issue #8258:
URL: https://github.com/apache/iceberg/issues/8258#issuecomment-1670547303
> My big thoughts on this is we should probably frame it as a full Spec Version update targeting version 3. Left some comments on the doc.
Yeah, I do think we should frame it in the new version. Let me update doc accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] advancedxy commented on issue #8258: multi-arg transform support
Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on issue #8258:
URL: https://github.com/apache/iceberg/issues/8258#issuecomment-1676610614
@RussellSpitzer I have update the spec change proposal doc. Would you mind to take a look again?
And I believe we should raise more discussion before committing more code, do you think it's necessary to post it in the dev mail list/slack to get more people's attention?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #8258: multi-arg transform support
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on issue #8258:
URL: https://github.com/apache/iceberg/issues/8258#issuecomment-1681188555
Yea I left some initial comments yesterday, plan to take another pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] multi-arg transform support [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8258:
URL: https://github.com/apache/iceberg/issues/8258#issuecomment-2347443909
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #8258: multi-arg transform support
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #8258:
URL: https://github.com/apache/iceberg/issues/8258#issuecomment-1680877561
@szehon-ho This is the proposal we were talking about, if you could check it out I would appreciate it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #8258: multi-arg transform support
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #8258:
URL: https://github.com/apache/iceberg/issues/8258#issuecomment-1670215126
My big thoughts on this is we should probably frame it as a full Spec Version update targeting version 3. Left some comments on the doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] multi-arg transform support [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #8258: multi-arg transform support
URL: https://github.com/apache/iceberg/issues/8258
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] multi-arg transform support [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8258:
URL: https://github.com/apache/iceberg/issues/8258#issuecomment-2378156864
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] advancedxy commented on issue #8258: multi-arg transform support
Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on issue #8258:
URL: https://github.com/apache/iceberg/issues/8258#issuecomment-1669795297
@RussellSpitzer @rdblue would you like to take a look at this? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #8258: multi-arg transform support
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #8258:
URL: https://github.com/apache/iceberg/issues/8258#issuecomment-1677164139
It definitely needs more eyes, it probably should get brought up at the next community sync as well Sent from my iPhoneOn Aug 13, 2023, at 10:16 PM, advancedxy ***@***.***> wrote:
@RussellSpitzer I have update the spec change proposal doc. Would you mind to take a look again?
And I believe we should raise more discussion before committing more code, do you think it's necessary to post it in the dev mail list/slack to get more people's attention?
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org