You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/11/13 12:59:24 UTC
[GitHub] [iceberg] ajantha-bhat opened a new issue #3547: Should we support sort order during table creation ?
ajantha-bhat opened a new issue #3547:
URL: https://github.com/apache/iceberg/issues/3547
In SQL create table syntax, when we create a table, should we support `sort order` in table properties?
I can see that only `ALTER TABLE ... WRITE ORDERED BY` is present and also we can sort files during `RewriteDataFiles`
will it be good to have it in create table as well to avoid calling alter table ?
If yes, please assign this issue to me. I can work on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on issue #3547: Should we support sort order during table creation SQL ?
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on issue #3547:
URL: https://github.com/apache/iceberg/issues/3547#issuecomment-1004292001
> then Spark should support it
+1
It's definitely achieveable in CREATE TABLE DDL, but I don't think using TBLPROPERTIES is a good idea. The table version upgrade makes sense to use a special property because we want all users in all engines to easily upgrade their tables to a newer version, and table properties is the best way as it exists in all versions of the CREATE TABLE DDL.
For sort order, different engines have different ways to define that. For example, in Hive DDL, there is a clause `CLUSTERED BY (col_name, col_name, ...) [SORTED BY (col_name [ASC|DESC], ...)] INTO num_buckets BUCKETS` , so if we really want to define sort order in create table, we should implement clauses like this on the engine side. (I am not saying to use the Hive DDL syntax in Spark, we should propose a syntax that could fully support Iceberg sort order features when we do it)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on issue #3547: Should we support sort order during table creation SQL ?
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #3547:
URL: https://github.com/apache/iceberg/issues/3547#issuecomment-1004201082
No, I don't think so. These aren't table properties and we don't want to rely on hacks like in `rewrite_data_files`. If we want this in DDL, then Spark should support it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat closed issue #3547: Should we support sort order during table creation SQL ?
Posted by GitBox <gi...@apache.org>.
ajantha-bhat closed issue #3547:
URL: https://github.com/apache/iceberg/issues/3547
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat edited a comment on issue #3547: Should we support sort order during table creation SQL ?
Posted by GitBox <gi...@apache.org>.
ajantha-bhat edited a comment on issue #3547:
URL: https://github.com/apache/iceberg/issues/3547#issuecomment-1003583744
@RussellSpitzer . @jackye1995 , @rdblue : Is it better to support sort order in the table properties (reserved property) and deprecate `ALTER TABLE ... WRITE ORDERED BY`?
So, that user can have sort order in the create table, can also alter using the `Alter table .. set tblproperties` and one less custom SQL in Iceberg?
P.s: I am planning to parse the sort order using spark default parser like I did for rewrite_data_files procedure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on issue #3547: Should we support sort order during table creation SQL ?
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3547:
URL: https://github.com/apache/iceberg/issues/3547#issuecomment-968193863
> I have a question. If sort is supportted with creation SQL, will it ensure the order of the whole table or only the current batch data? thank you.
It will act as a default sort order for the table. The insert's will use this sort order. However Iceberg also supports sort order evolution. So, when `ALTER TABLE ... WRITE ORDERED BY` it has to change table property also. When sort oder is changed, it won't impact previous written data. Only new data will follow new sort order.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hililiwei commented on issue #3547: Should we support sort order during table creation SQL ?
Posted by GitBox <gi...@apache.org>.
hililiwei commented on issue #3547:
URL: https://github.com/apache/iceberg/issues/3547#issuecomment-968191662
I have a question. If sort is supportted with creation SQL, will it ensure the order of the whole table or only the current batch data? thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on issue #3547: Should we support sort order during table creation SQL ?
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3547:
URL: https://github.com/apache/iceberg/issues/3547#issuecomment-1003583744
@RussellSpitzer . @jackye1995 , @rdblue : Is it better to support sort order in the table properties (reserved property) and deprecate `ALTER TABLE ... WRITE ORDERED BY`?
So, that user can have sort order in the create table and one less custom SQL in Iceberg?
P.s: I am planning to parse the sort order using spark default parser like I did for rewrite_data_files procedure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on issue #3547: Should we support sort order during table creation SQL ?
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3547:
URL: https://github.com/apache/iceberg/issues/3547#issuecomment-1004504733
ok. Thanks for the clarifications. I will close this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org