You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/05/25 11:19:19 UTC
[GitHub] [iceberg] wuwenchi opened a new pull request, #4865: Docs: add flink and iceberg type compatibility
wuwenchi opened a new pull request, #4865:
URL: https://github.com/apache/iceberg/pull/4865
Add flink and iceberg type compatibility.
Can you help review it? thanks !
@rdblue @openinx @kbendick
Closes #4864
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
samredai commented on PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#issuecomment-1170487509
Just FYI that there are no longer any subdirectories, so this markdown file is located at [docs/flink-getting-started.md](https://github.com/apache/iceberg/blob/master/docs/flink-getting-started.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wuwenchi commented on pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
wuwenchi commented on PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#issuecomment-1170668166
@samredai ok, thanks! @rdblue done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wuwenchi commented on a diff in pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
wuwenchi commented on code in PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#discussion_r882570740
##########
docs/flink/flink-getting-started.md:
##########
@@ -547,6 +547,64 @@ RewriteDataFilesActionResult result = Actions.forTable(table)
For more doc about options of the rewrite files action, please see [RewriteDataFilesAction](../../../javadoc/{{% icebergVersion %}}/org/apache/iceberg/flink/actions/RewriteDataFilesAction.html)
+## Type compatibility
+
+Flink and Iceberg support different set of types. Iceberg does the type conversion automatically, but not for all combinations,
Review Comment:
Good suggestion! I modified the description and added related notes, so that we can see some details more intuitively.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#discussion_r893006032
##########
docs/flink/flink-getting-started.md:
##########
@@ -547,6 +547,70 @@ RewriteDataFilesActionResult result = Actions.forTable(table)
For more doc about options of the rewrite files action, please see [RewriteDataFilesAction](../../../javadoc/{{% icebergVersion %}}/org/apache/iceberg/flink/actions/RewriteDataFilesAction.html)
+## Type conversion
+
+Iceberg's integration for Flink automatically converts between Flink and Iceberg types. When writing to a table with types that are not supported by Flink, like UUID, Iceberg will accept and convert values from the Flink type.
+
+### Flink to Iceberg
+
+Flink types are converted to Iceberg types according to the following table:
+
+| Flink | Iceberg | Notes |
+|----------------- |----------------------------|---------------|
+| boolean | boolean | |
+| tinyint | integer | |
+| smallint | integer | |
+| integer | integer | |
+| bigint | long | |
+| float | float | |
+| double | double | |
+| char | string | |
+| varchar | string | |
+| string | string | |
+| binary | binary | |
+| varbinary | fixed | |
+| decimal | decimal | |
+| date | date | |
+| time | time | |
+| timestamp | timestamp without timezone | |
+| timestamp_ltz | timestamp with timezone | |
+| array | list | |
+| map | map | |
+| multiset | map | |
+| row | struct | |
+| raw | | Not supported |
+| interval | | Not supported |
+| structured | | Not supported |
+| timestamp with zone | | Not supported |
+| distinct | | Not supported |
+| null | | Not supported |
+| symbol | | Not supported |
+| logical | | Not supported |
+
+### Iceberg to Flink
+
+Iceberg types are converted to Flink types according to the following table:
+
+| Iceberg | Flink | Notes |
+|----------------------------|---------------|--------------------------------------------------|
+| boolean | boolean | |
+| struct | row | |
+| list | array | |
+| map | map | |
+| integer | integer | |
+| long | bigint | |
+| float | float | |
+| double | double | |
+| date | date | |
+| time | time | |
+| timestamp without timezone | timestamp | precision is fixed at 6 |
+| timestamp with timezone | timestamp_ltz | precision is fixed at 6 |
+| string | varchar | length is fixed at 2<sup>31</sup>-1 |
+| uuid | binary | length is fixed at 16 |
Review Comment:
`binary(16)`?
##########
docs/flink/flink-getting-started.md:
##########
@@ -547,6 +547,70 @@ RewriteDataFilesActionResult result = Actions.forTable(table)
For more doc about options of the rewrite files action, please see [RewriteDataFilesAction](../../../javadoc/{{% icebergVersion %}}/org/apache/iceberg/flink/actions/RewriteDataFilesAction.html)
+## Type conversion
+
+Iceberg's integration for Flink automatically converts between Flink and Iceberg types. When writing to a table with types that are not supported by Flink, like UUID, Iceberg will accept and convert values from the Flink type.
+
+### Flink to Iceberg
+
+Flink types are converted to Iceberg types according to the following table:
+
+| Flink | Iceberg | Notes |
+|----------------- |----------------------------|---------------|
+| boolean | boolean | |
+| tinyint | integer | |
+| smallint | integer | |
+| integer | integer | |
+| bigint | long | |
+| float | float | |
+| double | double | |
+| char | string | |
+| varchar | string | |
+| string | string | |
+| binary | binary | |
+| varbinary | fixed | |
+| decimal | decimal | |
+| date | date | |
+| time | time | |
+| timestamp | timestamp without timezone | |
+| timestamp_ltz | timestamp with timezone | |
+| array | list | |
+| map | map | |
+| multiset | map | |
+| row | struct | |
+| raw | | Not supported |
+| interval | | Not supported |
+| structured | | Not supported |
+| timestamp with zone | | Not supported |
+| distinct | | Not supported |
+| null | | Not supported |
+| symbol | | Not supported |
+| logical | | Not supported |
+
+### Iceberg to Flink
+
+Iceberg types are converted to Flink types according to the following table:
+
+| Iceberg | Flink | Notes |
+|----------------------------|---------------|--------------------------------------------------|
+| boolean | boolean | |
+| struct | row | |
+| list | array | |
+| map | map | |
+| integer | integer | |
+| long | bigint | |
+| float | float | |
+| double | double | |
+| date | date | |
+| time | time | |
+| timestamp without timezone | timestamp | precision is fixed at 6 |
Review Comment:
`timestamp(6)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#discussion_r884324152
##########
docs/flink/flink-getting-started.md:
##########
@@ -547,6 +547,71 @@ RewriteDataFilesActionResult result = Actions.forTable(table)
For more doc about options of the rewrite files action, please see [RewriteDataFilesAction](../../../javadoc/{{% icebergVersion %}}/org/apache/iceberg/flink/actions/RewriteDataFilesAction.html)
+## Type compatibility
+
+Iceberg's type system is mapped to Flink's type system. This type conversion can be done by Iceberg automatically, though the following cases need to be considered.
+See the notes section below for compatibility concerns and how to overcome them.
+
+### Flink type to Iceberg type
+
+This type conversion table describes how Flink types are converted to the Iceberg types. The conversion applies on both creating Iceberg table and writing to Iceberg table via Flink.
+
+| Flink | Iceberg | Notes |
+|----------------- |----------------------------|---------------|
+| boolean | boolean | |
+| tinyint | integer | |
+| smallint | integer | |
+| integer | integer | |
+| bigint | long | |
+| float | float | |
+| double | double | |
+| char | string | |
+| varchar | string | |
+| string | string | |
+| binary | binary | |
+| varbinary | fixed | |
+| decimal | decimal | |
+| date | date | |
+| time | time | |
+| timestamp | timestamp without timezone | |
+| timestamp_ltz | timestamp with timezone | |
+| array | list | |
+| map | map | |
+| multiset | map | |
+| row | struct | |
+| raw | | Not supported |
+| interval | | Not supported |
+| structured | | Not supported |
+| timestamp with zone | | Not supported |
+| distinct | | Not supported |
+| null | | Not supported |
+| symbol | | Not supported |
+| logical | | Not supported |
+
+### Iceberg type to Flink type
+
+This type conversion table describes how Iceberg types are converted to the Flink types. The conversion applies on reading from Iceberg table via Flink.
+
+| Iceberg | Flink | Notes |
+|----------------------------|---------------|--------------------------------------------------|
+| boolean | boolean | |
+| struct | row | |
+| list | array | |
+| map | map | |
+| integer | integer | |
+| long | bigint | |
+| float | float | |
+| double | double | |
+| date | date | |
+| time | time | precision is fixed at 0 |
Review Comment:
precision is 0? I would expect that to be 6 for microseconds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
samredai commented on PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#issuecomment-1139746583
> There’s also an iceberg-docs repo, github.com/apache/iceberg-docs. You might need to make a PR there as well.
Since this is in the versioned part of the docs site, there's no need to open a PR against the iceberg-docs repo. I do think we need a bonafide "getting started"/"quickstart" flink guide that gives a docker environment and let's the user fully run a demo environment--that would go in the iceberg-docs repo.
On a separate note, I'm noticing there are a number of "Types Compatibility" subsections that are cropping up for various engines. I'm wondering if we should try and consolidate them to a single "Types Compatibility" page, or maybe append it to the Configurations page in PR #4801 (although it's not really 'configuration').
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wuwenchi commented on a diff in pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
wuwenchi commented on code in PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#discussion_r894086025
##########
docs/flink/flink-getting-started.md:
##########
@@ -547,6 +547,70 @@ RewriteDataFilesActionResult result = Actions.forTable(table)
For more doc about options of the rewrite files action, please see [RewriteDataFilesAction](../../../javadoc/{{% icebergVersion %}}/org/apache/iceberg/flink/actions/RewriteDataFilesAction.html)
+## Type conversion
+
+Iceberg's integration for Flink automatically converts between Flink and Iceberg types. When writing to a table with types that are not supported by Flink, like UUID, Iceberg will accept and convert values from the Flink type.
+
+### Flink to Iceberg
+
+Flink types are converted to Iceberg types according to the following table:
+
+| Flink | Iceberg | Notes |
+|----------------- |----------------------------|---------------|
+| boolean | boolean | |
+| tinyint | integer | |
+| smallint | integer | |
+| integer | integer | |
+| bigint | long | |
+| float | float | |
+| double | double | |
+| char | string | |
+| varchar | string | |
+| string | string | |
+| binary | binary | |
+| varbinary | fixed | |
+| decimal | decimal | |
+| date | date | |
+| time | time | |
+| timestamp | timestamp without timezone | |
+| timestamp_ltz | timestamp with timezone | |
+| array | list | |
+| map | map | |
+| multiset | map | |
+| row | struct | |
+| raw | | Not supported |
+| interval | | Not supported |
+| structured | | Not supported |
+| timestamp with zone | | Not supported |
+| distinct | | Not supported |
+| null | | Not supported |
+| symbol | | Not supported |
+| logical | | Not supported |
+
+### Iceberg to Flink
+
+Iceberg types are converted to Flink types according to the following table:
+
+| Iceberg | Flink | Notes |
+|----------------------------|---------------|--------------------------------------------------|
+| boolean | boolean | |
+| struct | row | |
+| list | array | |
+| map | map | |
+| integer | integer | |
+| long | bigint | |
+| float | float | |
+| double | double | |
+| date | date | |
+| time | time | |
+| timestamp without timezone | timestamp | precision is fixed at 6 |
+| timestamp with timezone | timestamp_ltz | precision is fixed at 6 |
+| string | varchar | length is fixed at 2<sup>31</sup>-1 |
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wuwenchi commented on a diff in pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
wuwenchi commented on code in PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#discussion_r884387712
##########
docs/flink/flink-getting-started.md:
##########
@@ -547,6 +547,70 @@ RewriteDataFilesActionResult result = Actions.forTable(table)
For more doc about options of the rewrite files action, please see [RewriteDataFilesAction](../../../javadoc/{{% icebergVersion %}}/org/apache/iceberg/flink/actions/RewriteDataFilesAction.html)
+## Type conversion
+
+Iceberg's integration for Flink automatically converts between Flink and Iceberg types. When writing to a table with types that are not supported by Flink, like UUID, Iceberg will accept and convert values from the Flink type.
+
+### Flink to Iceberg
+
+Flink types are converted to Iceberg types according to the following table:
+
+| Flink | Iceberg | Notes |
+|----------------- |----------------------------|---------------|
+| boolean | boolean | |
+| tinyint | integer | |
+| smallint | integer | |
+| integer | integer | |
+| bigint | long | |
+| float | float | |
+| double | double | |
+| char | string | |
+| varchar | string | |
+| string | string | |
+| binary | binary | |
+| varbinary | fixed | |
+| decimal | decimal | |
+| date | date | |
+| time | time | |
+| timestamp | timestamp without timezone | |
+| timestamp_ltz | timestamp with timezone | |
+| array | list | |
+| map | map | |
+| multiset | map | |
+| row | struct | |
+| raw | | Not supported |
+| interval | | Not supported |
+| structured | | Not supported |
+| timestamp with zone | | Not supported |
+| distinct | | Not supported |
+| null | | Not supported |
+| symbol | | Not supported |
+| logical | | Not supported |
+
+### Iceberg type to Flink type
Review Comment:
Should the `type` be removed here and be consistent with the above?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a diff in pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
kbendick commented on code in PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#discussion_r882341229
##########
docs/flink/flink-getting-started.md:
##########
@@ -547,6 +547,64 @@ RewriteDataFilesActionResult result = Actions.forTable(table)
For more doc about options of the rewrite files action, please see [RewriteDataFilesAction](../../../javadoc/{{% icebergVersion %}}/org/apache/iceberg/flink/actions/RewriteDataFilesAction.html)
+## Type compatibility
+
+Flink and Iceberg support different set of types. Iceberg does the type conversion automatically, but not for all combinations,
Review Comment:
Additionally, I would say they support “different” types. There are logic types (eg UUID), physical types (eg String or VARCHAR), and then what’s ultimately stored within the respective file types and how.
I’d focus on covering the first two here (logical and physical). And making notes of any peculiarities that are different within the matrix - for example, I believe that timestamp precision and what is and us not possible to store _might_ be worth calling more attention to in the notes section of the matrix.
For this sentence, I’d say something more along the lines of “Iceberg’s type system is mapped to Flink’s type system. This type conversion can be done by Iceberg automatically, though the following cases need to be considered. See the notes section below for compatibility concerns and how to overcome them”.
Then I’d make a small section listing the combinations that don’t work or general discussion of the mounts of concern. As we’re migrating the docs somewhat to a new structure, so I think that would really help (even if you can’t list all cases etc)
Just a suggestion / jumping off point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wuwenchi commented on a diff in pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
wuwenchi commented on code in PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#discussion_r882567429
##########
docs/flink/flink-getting-started.md:
##########
@@ -547,6 +547,64 @@ RewriteDataFilesActionResult result = Actions.forTable(table)
For more doc about options of the rewrite files action, please see [RewriteDataFilesAction](../../../javadoc/{{% icebergVersion %}}/org/apache/iceberg/flink/actions/RewriteDataFilesAction.html)
+## Type compatibility
+
+Flink and Iceberg support different set of types. Iceberg does the type conversion automatically, but not for all combinations,
+so you may want to understand the type conversion in Iceberg in prior to design the types of columns in your tables.
+
+### Flink type to Iceberg type
+
+This type conversion table describes how Spark types are converted to the Iceberg types. The conversion applies on both creating Iceberg table and writing to Iceberg table via Spark.
Review Comment:
=.= copy past issue... fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wuwenchi commented on a diff in pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
wuwenchi commented on code in PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#discussion_r894086157
##########
docs/flink/flink-getting-started.md:
##########
@@ -547,6 +547,70 @@ RewriteDataFilesActionResult result = Actions.forTable(table)
For more doc about options of the rewrite files action, please see [RewriteDataFilesAction](../../../javadoc/{{% icebergVersion %}}/org/apache/iceberg/flink/actions/RewriteDataFilesAction.html)
+## Type conversion
+
+Iceberg's integration for Flink automatically converts between Flink and Iceberg types. When writing to a table with types that are not supported by Flink, like UUID, Iceberg will accept and convert values from the Flink type.
+
+### Flink to Iceberg
+
+Flink types are converted to Iceberg types according to the following table:
+
+| Flink | Iceberg | Notes |
+|----------------- |----------------------------|---------------|
+| boolean | boolean | |
+| tinyint | integer | |
+| smallint | integer | |
+| integer | integer | |
+| bigint | long | |
+| float | float | |
+| double | double | |
+| char | string | |
+| varchar | string | |
+| string | string | |
+| binary | binary | |
+| varbinary | fixed | |
+| decimal | decimal | |
+| date | date | |
+| time | time | |
+| timestamp | timestamp without timezone | |
+| timestamp_ltz | timestamp with timezone | |
+| array | list | |
+| map | map | |
+| multiset | map | |
+| row | struct | |
+| raw | | Not supported |
+| interval | | Not supported |
+| structured | | Not supported |
+| timestamp with zone | | Not supported |
+| distinct | | Not supported |
+| null | | Not supported |
+| symbol | | Not supported |
+| logical | | Not supported |
+
+### Iceberg to Flink
+
+Iceberg types are converted to Flink types according to the following table:
+
+| Iceberg | Flink | Notes |
+|----------------------------|---------------|--------------------------------------------------|
+| boolean | boolean | |
+| struct | row | |
+| list | array | |
+| map | map | |
+| integer | integer | |
+| long | bigint | |
+| float | float | |
+| double | double | |
+| date | date | |
+| time | time | |
+| timestamp without timezone | timestamp | precision is fixed at 6 |
+| timestamp with timezone | timestamp_ltz | precision is fixed at 6 |
+| string | varchar | length is fixed at 2<sup>31</sup>-1 |
+| uuid | binary | length is fixed at 16 |
+| fixed | binary | |
+| binary | varbinary | width is fixed at 2<sup>31</sup>-1 |
+| decimal | decimal | precision and scale are the same as the original |
Review Comment:
done, looks more clear
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a diff in pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
kbendick commented on code in PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#discussion_r882337535
##########
docs/flink/flink-getting-started.md:
##########
@@ -547,6 +547,64 @@ RewriteDataFilesActionResult result = Actions.forTable(table)
For more doc about options of the rewrite files action, please see [RewriteDataFilesAction](../../../javadoc/{{% icebergVersion %}}/org/apache/iceberg/flink/actions/RewriteDataFilesAction.html)
+## Type compatibility
+
+Flink and Iceberg support different set of types. Iceberg does the type conversion automatically, but not for all combinations,
Review Comment:
Question: for what combinations can Iceberg not do the conversion automatically? From the perspective of a new user, this might leave them with more questions than answers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wuwenchi commented on a diff in pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
wuwenchi commented on code in PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#discussion_r884387113
##########
docs/flink/flink-getting-started.md:
##########
@@ -547,6 +547,71 @@ RewriteDataFilesActionResult result = Actions.forTable(table)
For more doc about options of the rewrite files action, please see [RewriteDataFilesAction](../../../javadoc/{{% icebergVersion %}}/org/apache/iceberg/flink/actions/RewriteDataFilesAction.html)
+## Type compatibility
+
+Iceberg's type system is mapped to Flink's type system. This type conversion can be done by Iceberg automatically, though the following cases need to be considered.
+See the notes section below for compatibility concerns and how to overcome them.
+
+### Flink type to Iceberg type
+
+This type conversion table describes how Flink types are converted to the Iceberg types. The conversion applies on both creating Iceberg table and writing to Iceberg table via Flink.
+
+| Flink | Iceberg | Notes |
+|----------------- |----------------------------|---------------|
+| boolean | boolean | |
+| tinyint | integer | |
+| smallint | integer | |
+| integer | integer | |
+| bigint | long | |
+| float | float | |
+| double | double | |
+| char | string | |
+| varchar | string | |
+| string | string | |
+| binary | binary | |
+| varbinary | fixed | |
+| decimal | decimal | |
+| date | date | |
+| time | time | |
+| timestamp | timestamp without timezone | |
+| timestamp_ltz | timestamp with timezone | |
+| array | list | |
+| map | map | |
+| multiset | map | |
+| row | struct | |
+| raw | | Not supported |
+| interval | | Not supported |
+| structured | | Not supported |
+| timestamp with zone | | Not supported |
+| distinct | | Not supported |
+| null | | Not supported |
+| symbol | | Not supported |
+| logical | | Not supported |
+
+### Iceberg type to Flink type
+
+This type conversion table describes how Iceberg types are converted to the Flink types. The conversion applies on reading from Iceberg table via Flink.
+
+| Iceberg | Flink | Notes |
+|----------------------------|---------------|--------------------------------------------------|
+| boolean | boolean | |
+| struct | row | |
+| list | array | |
+| map | map | |
+| integer | integer | |
+| long | bigint | |
+| float | float | |
+| double | double | |
+| date | date | |
+| time | time | precision is fixed at 0 |
Review Comment:
In flink, the precision of time is not supported and it default precision is 0:
```
in iceberg:
...
case TIME:
return new TimeType();
...
in flink:
public static final int DEFAULT_PRECISION = 0;
public TimeType() {
this(DEFAULT_PRECISION);
}
```
Of course, this is the behavior of flink, maybe we should not write this precision here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#discussion_r884949562
##########
docs/flink/flink-getting-started.md:
##########
@@ -547,6 +547,71 @@ RewriteDataFilesActionResult result = Actions.forTable(table)
For more doc about options of the rewrite files action, please see [RewriteDataFilesAction](../../../javadoc/{{% icebergVersion %}}/org/apache/iceberg/flink/actions/RewriteDataFilesAction.html)
+## Type compatibility
+
+Iceberg's type system is mapped to Flink's type system. This type conversion can be done by Iceberg automatically, though the following cases need to be considered.
+See the notes section below for compatibility concerns and how to overcome them.
+
+### Flink type to Iceberg type
+
+This type conversion table describes how Flink types are converted to the Iceberg types. The conversion applies on both creating Iceberg table and writing to Iceberg table via Flink.
+
+| Flink | Iceberg | Notes |
+|----------------- |----------------------------|---------------|
+| boolean | boolean | |
+| tinyint | integer | |
+| smallint | integer | |
+| integer | integer | |
+| bigint | long | |
+| float | float | |
+| double | double | |
+| char | string | |
+| varchar | string | |
+| string | string | |
+| binary | binary | |
+| varbinary | fixed | |
+| decimal | decimal | |
+| date | date | |
+| time | time | |
+| timestamp | timestamp without timezone | |
+| timestamp_ltz | timestamp with timezone | |
+| array | list | |
+| map | map | |
+| multiset | map | |
+| row | struct | |
+| raw | | Not supported |
+| interval | | Not supported |
+| structured | | Not supported |
+| timestamp with zone | | Not supported |
+| distinct | | Not supported |
+| null | | Not supported |
+| symbol | | Not supported |
+| logical | | Not supported |
+
+### Iceberg type to Flink type
+
+This type conversion table describes how Iceberg types are converted to the Flink types. The conversion applies on reading from Iceberg table via Flink.
+
+| Iceberg | Flink | Notes |
+|----------------------------|---------------|--------------------------------------------------|
+| boolean | boolean | |
+| struct | row | |
+| list | array | |
+| map | map | |
+| integer | integer | |
+| long | bigint | |
+| float | float | |
+| double | double | |
+| date | date | |
+| time | time | precision is fixed at 0 |
Review Comment:
> the precision of time is not supported
Then we should not mention it here.
##########
docs/flink/flink-getting-started.md:
##########
@@ -547,6 +547,70 @@ RewriteDataFilesActionResult result = Actions.forTable(table)
For more doc about options of the rewrite files action, please see [RewriteDataFilesAction](../../../javadoc/{{% icebergVersion %}}/org/apache/iceberg/flink/actions/RewriteDataFilesAction.html)
+## Type conversion
+
+Iceberg's integration for Flink automatically converts between Flink and Iceberg types. When writing to a table with types that are not supported by Flink, like UUID, Iceberg will accept and convert values from the Flink type.
+
+### Flink to Iceberg
+
+Flink types are converted to Iceberg types according to the following table:
+
+| Flink | Iceberg | Notes |
+|----------------- |----------------------------|---------------|
+| boolean | boolean | |
+| tinyint | integer | |
+| smallint | integer | |
+| integer | integer | |
+| bigint | long | |
+| float | float | |
+| double | double | |
+| char | string | |
+| varchar | string | |
+| string | string | |
+| binary | binary | |
+| varbinary | fixed | |
+| decimal | decimal | |
+| date | date | |
+| time | time | |
+| timestamp | timestamp without timezone | |
+| timestamp_ltz | timestamp with timezone | |
+| array | list | |
+| map | map | |
+| multiset | map | |
+| row | struct | |
+| raw | | Not supported |
+| interval | | Not supported |
+| structured | | Not supported |
+| timestamp with zone | | Not supported |
+| distinct | | Not supported |
+| null | | Not supported |
+| symbol | | Not supported |
+| logical | | Not supported |
+
+### Iceberg type to Flink type
Review Comment:
Yes, that would be good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
samredai commented on PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#issuecomment-1141264147
> I have another question, like #4725, the modifications in the spec should not need to submit a PR in iceberg-doc, right?
Yes that's right, the spec is copied over to iceberg-docs during the release process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wuwenchi commented on pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
wuwenchi commented on PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#issuecomment-1138432765
> There’s also an iceberg-docs repo, github.com/apache/iceberg-docs.
After this PR is completed, I will provide a PR to iceberg-docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a diff in pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
kbendick commented on code in PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#discussion_r882337173
##########
docs/flink/flink-getting-started.md:
##########
@@ -547,6 +547,64 @@ RewriteDataFilesActionResult result = Actions.forTable(table)
For more doc about options of the rewrite files action, please see [RewriteDataFilesAction](../../../javadoc/{{% icebergVersion %}}/org/apache/iceberg/flink/actions/RewriteDataFilesAction.html)
+## Type compatibility
+
+Flink and Iceberg support different set of types. Iceberg does the type conversion automatically, but not for all combinations,
+so you may want to understand the type conversion in Iceberg in prior to design the types of columns in your tables.
+
+### Flink type to Iceberg type
+
+This type conversion table describes how Spark types are converted to the Iceberg types. The conversion applies on both creating Iceberg table and writing to Iceberg table via Spark.
Review Comment:
Nit: this mentions Spark several times. Assuming a copy past issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wuwenchi commented on pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
wuwenchi commented on PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#issuecomment-1140925996
> Since this is in the versioned part of the docs site, there's no need to open a PR against the iceberg-docs repo.
I have another question, like #4725, the modifications in the spec should not need to submit a PR in iceberg-doc, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wuwenchi commented on pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
wuwenchi commented on PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#issuecomment-1140150284
@samredai
> I do think we need a bonafide "getting started"/"quickstart" flink guide that gives a docker environment and let's the user fully run a demo environment--that would go in the iceberg-docs repo.
Good idea, look forward to this!
> I'm wondering if we should try and consolidate them to a single "Types Compatibility" page, or maybe append it to the Configurations page
I prefer to add a new page to add this part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wuwenchi commented on a diff in pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
wuwenchi commented on code in PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#discussion_r894086083
##########
docs/flink/flink-getting-started.md:
##########
@@ -547,6 +547,70 @@ RewriteDataFilesActionResult result = Actions.forTable(table)
For more doc about options of the rewrite files action, please see [RewriteDataFilesAction](../../../javadoc/{{% icebergVersion %}}/org/apache/iceberg/flink/actions/RewriteDataFilesAction.html)
+## Type conversion
+
+Iceberg's integration for Flink automatically converts between Flink and Iceberg types. When writing to a table with types that are not supported by Flink, like UUID, Iceberg will accept and convert values from the Flink type.
+
+### Flink to Iceberg
+
+Flink types are converted to Iceberg types according to the following table:
+
+| Flink | Iceberg | Notes |
+|----------------- |----------------------------|---------------|
+| boolean | boolean | |
+| tinyint | integer | |
+| smallint | integer | |
+| integer | integer | |
+| bigint | long | |
+| float | float | |
+| double | double | |
+| char | string | |
+| varchar | string | |
+| string | string | |
+| binary | binary | |
+| varbinary | fixed | |
+| decimal | decimal | |
+| date | date | |
+| time | time | |
+| timestamp | timestamp without timezone | |
+| timestamp_ltz | timestamp with timezone | |
+| array | list | |
+| map | map | |
+| multiset | map | |
+| row | struct | |
+| raw | | Not supported |
+| interval | | Not supported |
+| structured | | Not supported |
+| timestamp with zone | | Not supported |
+| distinct | | Not supported |
+| null | | Not supported |
+| symbol | | Not supported |
+| logical | | Not supported |
+
+### Iceberg to Flink
+
+Iceberg types are converted to Flink types according to the following table:
+
+| Iceberg | Flink | Notes |
+|----------------------------|---------------|--------------------------------------------------|
+| boolean | boolean | |
+| struct | row | |
+| list | array | |
+| map | map | |
+| integer | integer | |
+| long | bigint | |
+| float | float | |
+| double | double | |
+| date | date | |
+| time | time | |
+| timestamp without timezone | timestamp | precision is fixed at 6 |
+| timestamp with timezone | timestamp_ltz | precision is fixed at 6 |
+| string | varchar | length is fixed at 2<sup>31</sup>-1 |
+| uuid | binary | length is fixed at 16 |
Review Comment:
done
##########
docs/flink/flink-getting-started.md:
##########
@@ -547,6 +547,70 @@ RewriteDataFilesActionResult result = Actions.forTable(table)
For more doc about options of the rewrite files action, please see [RewriteDataFilesAction](../../../javadoc/{{% icebergVersion %}}/org/apache/iceberg/flink/actions/RewriteDataFilesAction.html)
+## Type conversion
+
+Iceberg's integration for Flink automatically converts between Flink and Iceberg types. When writing to a table with types that are not supported by Flink, like UUID, Iceberg will accept and convert values from the Flink type.
+
+### Flink to Iceberg
+
+Flink types are converted to Iceberg types according to the following table:
+
+| Flink | Iceberg | Notes |
+|----------------- |----------------------------|---------------|
+| boolean | boolean | |
+| tinyint | integer | |
+| smallint | integer | |
+| integer | integer | |
+| bigint | long | |
+| float | float | |
+| double | double | |
+| char | string | |
+| varchar | string | |
+| string | string | |
+| binary | binary | |
+| varbinary | fixed | |
+| decimal | decimal | |
+| date | date | |
+| time | time | |
+| timestamp | timestamp without timezone | |
+| timestamp_ltz | timestamp with timezone | |
+| array | list | |
+| map | map | |
+| multiset | map | |
+| row | struct | |
+| raw | | Not supported |
+| interval | | Not supported |
+| structured | | Not supported |
+| timestamp with zone | | Not supported |
+| distinct | | Not supported |
+| null | | Not supported |
+| symbol | | Not supported |
+| logical | | Not supported |
+
+### Iceberg to Flink
+
+Iceberg types are converted to Flink types according to the following table:
+
+| Iceberg | Flink | Notes |
+|----------------------------|---------------|--------------------------------------------------|
+| boolean | boolean | |
+| struct | row | |
+| list | array | |
+| map | map | |
+| integer | integer | |
+| long | bigint | |
+| float | float | |
+| double | double | |
+| date | date | |
+| time | time | |
+| timestamp without timezone | timestamp | precision is fixed at 6 |
Review Comment:
done
##########
docs/flink/flink-getting-started.md:
##########
@@ -547,6 +547,70 @@ RewriteDataFilesActionResult result = Actions.forTable(table)
For more doc about options of the rewrite files action, please see [RewriteDataFilesAction](../../../javadoc/{{% icebergVersion %}}/org/apache/iceberg/flink/actions/RewriteDataFilesAction.html)
+## Type conversion
+
+Iceberg's integration for Flink automatically converts between Flink and Iceberg types. When writing to a table with types that are not supported by Flink, like UUID, Iceberg will accept and convert values from the Flink type.
+
+### Flink to Iceberg
+
+Flink types are converted to Iceberg types according to the following table:
+
+| Flink | Iceberg | Notes |
+|----------------- |----------------------------|---------------|
+| boolean | boolean | |
+| tinyint | integer | |
+| smallint | integer | |
+| integer | integer | |
+| bigint | long | |
+| float | float | |
+| double | double | |
+| char | string | |
+| varchar | string | |
+| string | string | |
+| binary | binary | |
+| varbinary | fixed | |
+| decimal | decimal | |
+| date | date | |
+| time | time | |
+| timestamp | timestamp without timezone | |
+| timestamp_ltz | timestamp with timezone | |
+| array | list | |
+| map | map | |
+| multiset | map | |
+| row | struct | |
+| raw | | Not supported |
+| interval | | Not supported |
+| structured | | Not supported |
+| timestamp with zone | | Not supported |
+| distinct | | Not supported |
+| null | | Not supported |
+| symbol | | Not supported |
+| logical | | Not supported |
+
+### Iceberg to Flink
+
+Iceberg types are converted to Flink types according to the following table:
+
+| Iceberg | Flink | Notes |
+|----------------------------|---------------|--------------------------------------------------|
+| boolean | boolean | |
+| struct | row | |
+| list | array | |
+| map | map | |
+| integer | integer | |
+| long | bigint | |
+| float | float | |
+| double | double | |
+| date | date | |
+| time | time | |
+| timestamp without timezone | timestamp | precision is fixed at 6 |
+| timestamp with timezone | timestamp_ltz | precision is fixed at 6 |
+| string | varchar | length is fixed at 2<sup>31</sup>-1 |
+| uuid | binary | length is fixed at 16 |
+| fixed | binary | |
+| binary | varbinary | width is fixed at 2<sup>31</sup>-1 |
Review Comment:
done
##########
docs/flink/flink-getting-started.md:
##########
@@ -547,6 +547,70 @@ RewriteDataFilesActionResult result = Actions.forTable(table)
For more doc about options of the rewrite files action, please see [RewriteDataFilesAction](../../../javadoc/{{% icebergVersion %}}/org/apache/iceberg/flink/actions/RewriteDataFilesAction.html)
+## Type conversion
+
+Iceberg's integration for Flink automatically converts between Flink and Iceberg types. When writing to a table with types that are not supported by Flink, like UUID, Iceberg will accept and convert values from the Flink type.
+
+### Flink to Iceberg
+
+Flink types are converted to Iceberg types according to the following table:
+
+| Flink | Iceberg | Notes |
+|----------------- |----------------------------|---------------|
+| boolean | boolean | |
+| tinyint | integer | |
+| smallint | integer | |
+| integer | integer | |
+| bigint | long | |
+| float | float | |
+| double | double | |
+| char | string | |
+| varchar | string | |
+| string | string | |
+| binary | binary | |
+| varbinary | fixed | |
+| decimal | decimal | |
+| date | date | |
+| time | time | |
+| timestamp | timestamp without timezone | |
+| timestamp_ltz | timestamp with timezone | |
+| array | list | |
+| map | map | |
+| multiset | map | |
+| row | struct | |
+| raw | | Not supported |
+| interval | | Not supported |
+| structured | | Not supported |
+| timestamp with zone | | Not supported |
+| distinct | | Not supported |
+| null | | Not supported |
+| symbol | | Not supported |
+| logical | | Not supported |
+
+### Iceberg to Flink
+
+Iceberg types are converted to Flink types according to the following table:
+
+| Iceberg | Flink | Notes |
+|----------------------------|---------------|--------------------------------------------------|
+| boolean | boolean | |
+| struct | row | |
+| list | array | |
+| map | map | |
+| integer | integer | |
+| long | bigint | |
+| float | float | |
+| double | double | |
+| date | date | |
+| time | time | |
+| timestamp without timezone | timestamp | precision is fixed at 6 |
+| timestamp with timezone | timestamp_ltz | precision is fixed at 6 |
+| string | varchar | length is fixed at 2<sup>31</sup>-1 |
+| uuid | binary | length is fixed at 16 |
+| fixed | binary | |
+| binary | varbinary | width is fixed at 2<sup>31</sup>-1 |
+| decimal | decimal | precision and scale are the same as the original |
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
rdblue merged PR #4865:
URL: https://github.com/apache/iceberg/pull/4865
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#discussion_r893006212
##########
docs/flink/flink-getting-started.md:
##########
@@ -547,6 +547,70 @@ RewriteDataFilesActionResult result = Actions.forTable(table)
For more doc about options of the rewrite files action, please see [RewriteDataFilesAction](../../../javadoc/{{% icebergVersion %}}/org/apache/iceberg/flink/actions/RewriteDataFilesAction.html)
+## Type conversion
+
+Iceberg's integration for Flink automatically converts between Flink and Iceberg types. When writing to a table with types that are not supported by Flink, like UUID, Iceberg will accept and convert values from the Flink type.
+
+### Flink to Iceberg
+
+Flink types are converted to Iceberg types according to the following table:
+
+| Flink | Iceberg | Notes |
+|----------------- |----------------------------|---------------|
+| boolean | boolean | |
+| tinyint | integer | |
+| smallint | integer | |
+| integer | integer | |
+| bigint | long | |
+| float | float | |
+| double | double | |
+| char | string | |
+| varchar | string | |
+| string | string | |
+| binary | binary | |
+| varbinary | fixed | |
+| decimal | decimal | |
+| date | date | |
+| time | time | |
+| timestamp | timestamp without timezone | |
+| timestamp_ltz | timestamp with timezone | |
+| array | list | |
+| map | map | |
+| multiset | map | |
+| row | struct | |
+| raw | | Not supported |
+| interval | | Not supported |
+| structured | | Not supported |
+| timestamp with zone | | Not supported |
+| distinct | | Not supported |
+| null | | Not supported |
+| symbol | | Not supported |
+| logical | | Not supported |
+
+### Iceberg to Flink
+
+Iceberg types are converted to Flink types according to the following table:
+
+| Iceberg | Flink | Notes |
+|----------------------------|---------------|--------------------------------------------------|
+| boolean | boolean | |
+| struct | row | |
+| list | array | |
+| map | map | |
+| integer | integer | |
+| long | bigint | |
+| float | float | |
+| double | double | |
+| date | date | |
+| time | time | |
+| timestamp without timezone | timestamp | precision is fixed at 6 |
+| timestamp with timezone | timestamp_ltz | precision is fixed at 6 |
+| string | varchar | length is fixed at 2<sup>31</sup>-1 |
+| uuid | binary | length is fixed at 16 |
+| fixed | binary | |
+| binary | varbinary | width is fixed at 2<sup>31</sup>-1 |
Review Comment:
`varbinary(2147483647)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue closed pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
rdblue closed pull request #4865: Docs: add flink and iceberg type compatibility
URL: https://github.com/apache/iceberg/pull/4865
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wuwenchi commented on a diff in pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
wuwenchi commented on code in PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#discussion_r885143325
##########
docs/flink/flink-getting-started.md:
##########
@@ -547,6 +547,71 @@ RewriteDataFilesActionResult result = Actions.forTable(table)
For more doc about options of the rewrite files action, please see [RewriteDataFilesAction](../../../javadoc/{{% icebergVersion %}}/org/apache/iceberg/flink/actions/RewriteDataFilesAction.html)
+## Type compatibility
+
+Iceberg's type system is mapped to Flink's type system. This type conversion can be done by Iceberg automatically, though the following cases need to be considered.
+See the notes section below for compatibility concerns and how to overcome them.
+
+### Flink type to Iceberg type
+
+This type conversion table describes how Flink types are converted to the Iceberg types. The conversion applies on both creating Iceberg table and writing to Iceberg table via Flink.
+
+| Flink | Iceberg | Notes |
+|----------------- |----------------------------|---------------|
+| boolean | boolean | |
+| tinyint | integer | |
+| smallint | integer | |
+| integer | integer | |
+| bigint | long | |
+| float | float | |
+| double | double | |
+| char | string | |
+| varchar | string | |
+| string | string | |
+| binary | binary | |
+| varbinary | fixed | |
+| decimal | decimal | |
+| date | date | |
+| time | time | |
+| timestamp | timestamp without timezone | |
+| timestamp_ltz | timestamp with timezone | |
+| array | list | |
+| map | map | |
+| multiset | map | |
+| row | struct | |
+| raw | | Not supported |
+| interval | | Not supported |
+| structured | | Not supported |
+| timestamp with zone | | Not supported |
+| distinct | | Not supported |
+| null | | Not supported |
+| symbol | | Not supported |
+| logical | | Not supported |
+
+### Iceberg type to Flink type
+
+This type conversion table describes how Iceberg types are converted to the Flink types. The conversion applies on reading from Iceberg table via Flink.
+
+| Iceberg | Flink | Notes |
+|----------------------------|---------------|--------------------------------------------------|
+| boolean | boolean | |
+| struct | row | |
+| list | array | |
+| map | map | |
+| integer | integer | |
+| long | bigint | |
+| float | float | |
+| double | double | |
+| date | date | |
+| time | time | precision is fixed at 0 |
Review Comment:
done
##########
docs/flink/flink-getting-started.md:
##########
@@ -547,6 +547,70 @@ RewriteDataFilesActionResult result = Actions.forTable(table)
For more doc about options of the rewrite files action, please see [RewriteDataFilesAction](../../../javadoc/{{% icebergVersion %}}/org/apache/iceberg/flink/actions/RewriteDataFilesAction.html)
+## Type conversion
+
+Iceberg's integration for Flink automatically converts between Flink and Iceberg types. When writing to a table with types that are not supported by Flink, like UUID, Iceberg will accept and convert values from the Flink type.
+
+### Flink to Iceberg
+
+Flink types are converted to Iceberg types according to the following table:
+
+| Flink | Iceberg | Notes |
+|----------------- |----------------------------|---------------|
+| boolean | boolean | |
+| tinyint | integer | |
+| smallint | integer | |
+| integer | integer | |
+| bigint | long | |
+| float | float | |
+| double | double | |
+| char | string | |
+| varchar | string | |
+| string | string | |
+| binary | binary | |
+| varbinary | fixed | |
+| decimal | decimal | |
+| date | date | |
+| time | time | |
+| timestamp | timestamp without timezone | |
+| timestamp_ltz | timestamp with timezone | |
+| array | list | |
+| map | map | |
+| multiset | map | |
+| row | struct | |
+| raw | | Not supported |
+| interval | | Not supported |
+| structured | | Not supported |
+| timestamp with zone | | Not supported |
+| distinct | | Not supported |
+| null | | Not supported |
+| symbol | | Not supported |
+| logical | | Not supported |
+
+### Iceberg type to Flink type
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#discussion_r893006310
##########
docs/flink/flink-getting-started.md:
##########
@@ -547,6 +547,70 @@ RewriteDataFilesActionResult result = Actions.forTable(table)
For more doc about options of the rewrite files action, please see [RewriteDataFilesAction](../../../javadoc/{{% icebergVersion %}}/org/apache/iceberg/flink/actions/RewriteDataFilesAction.html)
+## Type conversion
+
+Iceberg's integration for Flink automatically converts between Flink and Iceberg types. When writing to a table with types that are not supported by Flink, like UUID, Iceberg will accept and convert values from the Flink type.
+
+### Flink to Iceberg
+
+Flink types are converted to Iceberg types according to the following table:
+
+| Flink | Iceberg | Notes |
+|----------------- |----------------------------|---------------|
+| boolean | boolean | |
+| tinyint | integer | |
+| smallint | integer | |
+| integer | integer | |
+| bigint | long | |
+| float | float | |
+| double | double | |
+| char | string | |
+| varchar | string | |
+| string | string | |
+| binary | binary | |
+| varbinary | fixed | |
+| decimal | decimal | |
+| date | date | |
+| time | time | |
+| timestamp | timestamp without timezone | |
+| timestamp_ltz | timestamp with timezone | |
+| array | list | |
+| map | map | |
+| multiset | map | |
+| row | struct | |
+| raw | | Not supported |
+| interval | | Not supported |
+| structured | | Not supported |
+| timestamp with zone | | Not supported |
+| distinct | | Not supported |
+| null | | Not supported |
+| symbol | | Not supported |
+| logical | | Not supported |
+
+### Iceberg to Flink
+
+Iceberg types are converted to Flink types according to the following table:
+
+| Iceberg | Flink | Notes |
+|----------------------------|---------------|--------------------------------------------------|
+| boolean | boolean | |
+| struct | row | |
+| list | array | |
+| map | map | |
+| integer | integer | |
+| long | bigint | |
+| float | float | |
+| double | double | |
+| date | date | |
+| time | time | |
+| timestamp without timezone | timestamp | precision is fixed at 6 |
+| timestamp with timezone | timestamp_ltz | precision is fixed at 6 |
+| string | varchar | length is fixed at 2<sup>31</sup>-1 |
+| uuid | binary | length is fixed at 16 |
+| fixed | binary | |
+| binary | varbinary | width is fixed at 2<sup>31</sup>-1 |
+| decimal | decimal | precision and scale are the same as the original |
Review Comment:
`decimal(P, S)` <=> `decimal(P, S)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#discussion_r893005955
##########
docs/flink/flink-getting-started.md:
##########
@@ -547,6 +547,70 @@ RewriteDataFilesActionResult result = Actions.forTable(table)
For more doc about options of the rewrite files action, please see [RewriteDataFilesAction](../../../javadoc/{{% icebergVersion %}}/org/apache/iceberg/flink/actions/RewriteDataFilesAction.html)
+## Type conversion
+
+Iceberg's integration for Flink automatically converts between Flink and Iceberg types. When writing to a table with types that are not supported by Flink, like UUID, Iceberg will accept and convert values from the Flink type.
+
+### Flink to Iceberg
+
+Flink types are converted to Iceberg types according to the following table:
+
+| Flink | Iceberg | Notes |
+|----------------- |----------------------------|---------------|
+| boolean | boolean | |
+| tinyint | integer | |
+| smallint | integer | |
+| integer | integer | |
+| bigint | long | |
+| float | float | |
+| double | double | |
+| char | string | |
+| varchar | string | |
+| string | string | |
+| binary | binary | |
+| varbinary | fixed | |
+| decimal | decimal | |
+| date | date | |
+| time | time | |
+| timestamp | timestamp without timezone | |
+| timestamp_ltz | timestamp with timezone | |
+| array | list | |
+| map | map | |
+| multiset | map | |
+| row | struct | |
+| raw | | Not supported |
+| interval | | Not supported |
+| structured | | Not supported |
+| timestamp with zone | | Not supported |
+| distinct | | Not supported |
+| null | | Not supported |
+| symbol | | Not supported |
+| logical | | Not supported |
+
+### Iceberg to Flink
+
+Iceberg types are converted to Flink types according to the following table:
+
+| Iceberg | Flink | Notes |
+|----------------------------|---------------|--------------------------------------------------|
+| boolean | boolean | |
+| struct | row | |
+| list | array | |
+| map | map | |
+| integer | integer | |
+| long | bigint | |
+| float | float | |
+| double | double | |
+| date | date | |
+| time | time | |
+| timestamp without timezone | timestamp | precision is fixed at 6 |
+| timestamp with timezone | timestamp_ltz | precision is fixed at 6 |
+| string | varchar | length is fixed at 2<sup>31</sup>-1 |
Review Comment:
Why not use `varchar[2147483647]`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
rdblue commented on PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#issuecomment-1170328509
@wuwenchi, can you rebase?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #4865: Docs: add flink and iceberg type compatibility
Posted by GitBox <gi...@apache.org>.
rdblue commented on PR #4865:
URL: https://github.com/apache/iceberg/pull/4865#issuecomment-1173165553
Thanks, @wuwenchi!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org