You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/07/13 14:52:51 UTC
[GitHub] [iceberg] XuQianJin-Stars opened a new pull request #2818: Convert Type for update schema
XuQianJin-Stars opened a new pull request #2818:
URL: https://github.com/apache/iceberg/pull/2818
Add table format conversion type conversion
<p class="auto-cursor-target" style="margin: 10px 0px 0px; padding: 0px; color: rgb(23, 43, 77); font-family: -apple-system, system-ui, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Type conversion currently supported by hive：</p><div class="table-wrap" style="margin: 10px 0px 0px; padding: 0px; overflow-x: auto; color: rgb(23, 43, 77); font-family: -apple-system, system-ui, "Segoe UI", Roboto, Oxygen, Ubuntu, "Fira Sans", "Droid Sans", &quo
t;Helvetica Neue", sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">
| boolean | tinyint | smallint | int | bigint | float | double | decimal | string | varchar | timestamp | date | binary
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
boolean | true | false | false | false | false | false | false | false | false | false | false | false | false
tinyint | false | true | true | true | true | true | true | true | true | true | false | false | false
smallint | false | false | true | true | true | true | true | true | true | true | false | false | false
int | false | false | false | true | true | true | true | true | true | true | false | false | false
bigint | false | false | false | false | true | true | true | true | true | true | false | false | false
float | false | false | false | false | false | true | true | true | true | true | false | false | false
double | false | false | false | false | false | false | true | true | true | true | false | false | false
decimal | false | false | false | false | false | false | false | true | true | true | false | false | false
string | false | false | false | false | false | false | true | true | true | true | false | false | false
varchar | false | false | false | false | false | false | true | true | true | true | false | false | false
timestamp | false | false | false | false | false | false | false | false | true | true | true | false | false
date | false | false | false | false | false | false | false | false | true | true | false | true | false
binary | false | false | false | false | false | false | false | false | false | false | false | false | true
</div>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jerryshao edited a comment on pull request #2818: Core: Convert Type for update schema
Posted by GitBox <gi...@apache.org>.
jerryshao edited a comment on pull request #2818:
URL: https://github.com/apache/iceberg/pull/2818#issuecomment-884186031
I would suggest to add the current compatibility matrix to docs firstly, then bring out a discussion in dev list, the changes here may bring in compatibility inconsistency, it would be better to have a discussion at first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] XuQianJin-Stars commented on pull request #2818: Core: Convert Type for update schema
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on pull request #2818:
URL: https://github.com/apache/iceberg/pull/2818#issuecomment-881836475
> I think the reason Iceberg only does the 3 conversions now is because of partition spec compatibility. For example, if a column `id` is of int type, and has a partition field `bucket(16, id)`, and you change it to float type, then the partition transformation would fail.
>
> I think it is a very great idea to have a matrix for compatibility, but we need to check one by one to see if any partition transform would break, and have a matrix for Iceberg.
>
> In addition, allowing too many update options would limit our ability to introduce new partition transform functions. I don't see any new ones added soon, but I don't know if anyone has some thinking about some other potentially beneficial transforms to have in long run that we should discuss. Probably worth brainstorming in the dev list.
Well, in fact, there are already some implicit conversions on the computing engine side, and both spark and flink have some conventional support. If we need to cover all conversion support, we also need to add some test verification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jerryshao edited a comment on pull request #2818: Core: Convert Type for update schema
Posted by GitBox <gi...@apache.org>.
jerryshao edited a comment on pull request #2818:
URL: https://github.com/apache/iceberg/pull/2818#issuecomment-884186031
I would suggest to add the current compatibility matrix to docs firstly, then bring out a discussion in dev list. The changes here may bring in compatibility inconsistency, it would be better to have a discussion at first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jerryshao commented on pull request #2818: Core: Convert Type for update schema
Posted by GitBox <gi...@apache.org>.
jerryshao commented on pull request #2818:
URL: https://github.com/apache/iceberg/pull/2818#issuecomment-884186031
I would suggest to add the compatibility matrix to docs firstly, then bring out a discussion in dev list, the changes here may bring in compatibility inconsistency, it would be better to have a discussion at first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on pull request #2818: Core: Convert Type for update schema
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on pull request #2818:
URL: https://github.com/apache/iceberg/pull/2818#issuecomment-881642743
I think the reason Iceberg only does the 3 conversions now is because of partition spec compatibility. For example, if a column `id` is of int type, and has a partition field `bucket(16, id)`, and you change it to float type, then the partition transformation would fail.
I think it is a very great idea to have a matrix for compatibility, but we need to check one by one to see if any partition transform would break, and have a matrix for Iceberg.
In addition, allowing too many update options would limit our ability to introduce new partition transform functions. I don't see any new ones added soon, but I don't know if anyone has some thinking about some other potentially beneficial transforms to have in long run that we should discuss. Probably worth brainstorming in the dev list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org