You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/12/30 14:05:50 UTC
[GitHub] [iceberg] xiaotianzhang01 opened a new pull request #3827: Docs: Add `Streaming upsert write` in flink.md
xiaotianzhang01 opened a new pull request #3827:
URL: https://github.com/apache/iceberg/pull/3827
Add a separate use case of `write.upsert.enabled` in the flink document to answer some questions of flink users. Is this necessary?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Reo-LEI commented on a change in pull request #3827: Docs: Add `Streaming upsert write` in flink.md
Posted by GitBox <gi...@apache.org>.
Reo-LEI commented on a change in pull request #3827:
URL: https://github.com/apache/iceberg/pull/3827#discussion_r782761684
##########
File path: site/docs/flink.md
##########
@@ -522,6 +522,20 @@ FlinkSink.forRowData(input)
env.execute("Test Iceberg DataStream");
```
+## Streaming upsert write
+
+Iceberg supports upsert based on the primary key when writing streaming data into v2 table format.
+
+```sql
+CREATE TABLE `hive_catalog`.`default`.`sample` (
+ `id` INT UNIQUE COMMENT 'unique id',
+ `data` STRING NOT NULL,
+ PRIMARY KEY(`id`) NOT ENFORCED
+) with ('format-version'='2', 'write.upsert.enabled'='true');
+```
+!!! Note
Review comment:
In UPSERT mode, if the table is partitioned, the partition fields should be a part of equality fields. I think this is important and should be mentioned in the docs.
https://github.com/apache/iceberg/blob/e0ff71dfa8ffb54941fcc82ae127ab06f271331c/flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java#L378
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] xiaotianzhang01 commented on a change in pull request #3827: Docs: Add `Streaming upsert write` in flink.md
Posted by GitBox <gi...@apache.org>.
xiaotianzhang01 commented on a change in pull request #3827:
URL: https://github.com/apache/iceberg/pull/3827#discussion_r782853135
##########
File path: site/docs/flink.md
##########
@@ -522,6 +522,20 @@ FlinkSink.forRowData(input)
env.execute("Test Iceberg DataStream");
```
+## Streaming upsert write
+
+Iceberg supports upsert based on the primary key when writing streaming data into v2 table format.
+
+```sql
+CREATE TABLE `hive_catalog`.`default`.`sample` (
+ `id` INT UNIQUE COMMENT 'unique id',
+ `data` STRING NOT NULL,
+ PRIMARY KEY(`id`) NOT ENFORCED
+) with ('format-version'='2', 'write.upsert.enabled'='true');
+```
+!!! Note
Review comment:
okey
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] xiaotianzhang01 commented on pull request #3827: Docs: Add `Streaming upsert write` in flink.md
Posted by GitBox <gi...@apache.org>.
xiaotianzhang01 commented on pull request #3827:
URL: https://github.com/apache/iceberg/pull/3827#issuecomment-1011825295
CC: @openinx @Reo-LEI @rdblue @jackye1995 @RussellSpitzer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] xiaotianzhang01 commented on a change in pull request #3827: Docs: Add `Streaming upsert write` in flink.md
Posted by GitBox <gi...@apache.org>.
xiaotianzhang01 commented on a change in pull request #3827:
URL: https://github.com/apache/iceberg/pull/3827#discussion_r778644184
##########
File path: site/docs/flink.md
##########
@@ -522,6 +522,18 @@ FlinkSink.forRowData(input)
env.execute("Test Iceberg DataStream");
```
+## Streaming upsert write
+
+Iceberg supports upsert based on the primary key when writing streaming data.
Review comment:
Thank you for your reply, this can be removed. The introduction of this feature is needed for new users of iceberg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on a change in pull request #3827: Docs: Add `Streaming upsert write` in flink.md
Posted by GitBox <gi...@apache.org>.
samredai commented on a change in pull request #3827:
URL: https://github.com/apache/iceberg/pull/3827#discussion_r777481886
##########
File path: site/docs/flink.md
##########
@@ -522,6 +522,18 @@ FlinkSink.forRowData(input)
env.execute("Test Iceberg DataStream");
```
+## Streaming upsert write
+
+Iceberg supports upsert based on the primary key when writing streaming data.
Review comment:
Would it be helpful to state here that this requires v2 tables?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] xiaotianzhang01 commented on a change in pull request #3827: Docs: Add `Streaming upsert write` in flink.md
Posted by GitBox <gi...@apache.org>.
xiaotianzhang01 commented on a change in pull request #3827:
URL: https://github.com/apache/iceberg/pull/3827#discussion_r778682794
##########
File path: site/docs/flink.md
##########
@@ -522,6 +522,18 @@ FlinkSink.forRowData(input)
env.execute("Test Iceberg DataStream");
```
+## Streaming upsert write
+
+Iceberg supports upsert based on the primary key when writing streaming data.
Review comment:
yeap, update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] xiaotianzhang01 commented on a change in pull request #3827: Docs: Add `Streaming upsert write` in flink.md
Posted by GitBox <gi...@apache.org>.
xiaotianzhang01 commented on a change in pull request #3827:
URL: https://github.com/apache/iceberg/pull/3827#discussion_r778644184
##########
File path: site/docs/flink.md
##########
@@ -522,6 +522,18 @@ FlinkSink.forRowData(input)
env.execute("Test Iceberg DataStream");
```
+## Streaming upsert write
+
+Iceberg supports upsert based on the primary key when writing streaming data.
Review comment:
Thank you for your reply, this can be removed. The introduction of this feature is needed for new users of iceberg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Reo-LEI commented on pull request #3827: Docs: Add `Streaming upsert write` in flink.md
Posted by GitBox <gi...@apache.org>.
Reo-LEI commented on pull request #3827:
URL: https://github.com/apache/iceberg/pull/3827#issuecomment-1011837361
LGTM, Thanks for your contribution! @xiaotianzhang01
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org