You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/11/24 02:34:01 UTC
[GitHub] [iceberg] yinshan612 opened a new issue, #6262: Does iceberg not support stream reading and updating tables
yinshan612 opened a new issue, #6262:
URL: https://github.com/apache/iceberg/issues/6262
### Query engine
Flink
### Question
Read the test table through the iceberg stream, and then update the test. The stream read task will report an error and terminate. Is this a bug or a function not implemented at present
一.sqlclient
CREATE TABLE test2(
id BIGINT COMMENT 'unique id',
data STRING,
primary key(id) not ENFORCED
) WITH (
'connector'='iceberg',
'catalog-name'='hadoop_catalog',
'catalog-type'='hadoop',
'warehouse'='hdfs://nn1:8020/warehouse/iceberg1',
'format-version'='2'
);
insert into test2 values (1,'202-11-20');
insert into test2 values (1,'2022-11-11');
二.other sqlclient
select * from test2 OPTIONS('streaming'='true', 'monitor-interval'='10s');
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra closed issue #6262: Does iceberg not support stream reading and updating tables
Posted by GitBox <gi...@apache.org>.
nastra closed issue #6262: Does iceberg not support stream reading and updating tables
URL: https://github.com/apache/iceberg/issues/6262
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yinshan612 commented on issue #6262: Does iceberg not support stream reading and updating tables
Posted by GitBox <gi...@apache.org>.
yinshan612 commented on issue #6262:
URL: https://github.com/apache/iceberg/issues/6262#issuecomment-1326925358
> Correct, via SQL + streaming you can only select + append, see also https://iceberg.apache.org/docs/latest/flink/#flink for what's supported currently
Will you support this in the future
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on issue #6262: Does iceberg not support stream reading and updating tables
Posted by GitBox <gi...@apache.org>.
nastra commented on issue #6262:
URL: https://github.com/apache/iceberg/issues/6262#issuecomment-1326163682
@yinshan612 could you please attach the error/stacktrace you're seeing in this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on issue #6262: Does iceberg not support stream reading and updating tables
Posted by GitBox <gi...@apache.org>.
nastra commented on issue #6262:
URL: https://github.com/apache/iceberg/issues/6262#issuecomment-1327088674
/cc @stevenzwu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on issue #6262: Does iceberg not support stream reading and updating tables
Posted by GitBox <gi...@apache.org>.
nastra commented on issue #6262:
URL: https://github.com/apache/iceberg/issues/6262#issuecomment-1326203246
I don't think it's supported to perform **overwrites** when streaming is enabled. When streaming is enabled, Iceberg will use [Incremental scans](https://github.com/apache/iceberg/blob/master/flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/source/FlinkSplitPlanner.java#L123) and those do not support [overwrites](https://github.com/apache/iceberg/blob/8b8a1038609610873c7d40023b4fc98ba341a502/core/src/main/java/org/apache/iceberg/IncrementalDataTableScan.java#L140-L145).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on issue #6262: Does iceberg not support stream reading and updating tables
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on issue #6262:
URL: https://github.com/apache/iceberg/issues/6262#issuecomment-1327666303
@yinshan612 support for deletes will probably be added in the future. e.g., here is one PR from @Reo-LEI
https://github.com/apache/iceberg/pull/6182/files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yinshan612 commented on issue #6262: Does iceberg not support stream reading and updating tables
Posted by GitBox <gi...@apache.org>.
yinshan612 commented on issue #6262:
URL: https://github.com/apache/iceberg/issues/6262#issuecomment-1327977589
> @yinshan612 support for deletes will probably be added in the future. e.g., here is one PR from @Reo-LEI https://github.com/apache/iceberg/pull/6182/files
okay, thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on issue #6262: Does iceberg not support stream reading and updating tables
Posted by GitBox <gi...@apache.org>.
nastra commented on issue #6262:
URL: https://github.com/apache/iceberg/issues/6262#issuecomment-1326252195
Correct, via SQL + streaming you can only select + append, see also https://iceberg.apache.org/docs/latest/flink/#flink for what's supported currently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yinshan612 commented on issue #6262: Does iceberg not support stream reading and updating tables
Posted by GitBox <gi...@apache.org>.
yinshan612 commented on issue #6262:
URL: https://github.com/apache/iceberg/issues/6262#issuecomment-1326171782
>
The first insert is OK, but the second insert of the same key is the update operation. At this time, the following error will be reported when the stream of iceberg reads the table
error: Found overwrite operation, cannot support incremental data in snapshots (4030445746041511082, 4197372471984049664]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yinshan612 commented on issue #6262: Does iceberg not support stream reading and updating tables
Posted by GitBox <gi...@apache.org>.
yinshan612 commented on issue #6262:
URL: https://github.com/apache/iceberg/issues/6262#issuecomment-1326214219
> Collaborator
It means that when monitoring a table through the stream read of iceberg, the table can only be append, not update or delete. Am I right in understanding this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org