You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/12/01 09:14:46 UTC
[GitHub] [iceberg] lichaohao opened a new issue, #6330: iceberg : format-version=2 , when the job is running (insert and update), can not execute rewrite small data file ?
lichaohao opened a new issue, #6330:
URL: https://github.com/apache/iceberg/issues/6330
### Query engine
iceberg:1.0.0
spark:3.2.0
flink:1.13.2
catalog:hive-catalog
### Question
iceberg:1.0.0
spark:3.2.0
flink:1.13.2
catalog:hive-catalog
source table:mysql cdc table: mysql_cdc_source
sink table:iceberg table: my_iceberg_sink ==> primary key id ,format-version=2, write.upsert.enabled=true
execute sql: (checkpoint 1min)
upsert into my_iceberg_sink select * from mysql_cdc_source;
ps: mysql exists insert and update operation
when the job is running somet time, i want to rewrite the iceberg data file into bigger one, 【spark execute】 (call "hive_prod.system.rewrite_data_files(my_iceberg_sink)")
get the following exception message:
can not commit,found new position delete for replaced data file: GenericDataFile....hdfs://xxxxdata.parquet
all above that what should i do can execute "call hive_prod.system.rewrite_data_files(my_iceberg_sink) " correctly?
Thank you for your answer！
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] commented on issue #6330: iceberg : format-version=2 , when the job is running (insert and update), can not execute rewrite small data file ?
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6330:
URL: https://github.com/apache/iceberg/issues/6330#issuecomment-1574462455
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] SHuixo commented on issue #6330: iceberg : format-version=2 , when the job is running (insert and update), can not execute rewrite small data file ?
Posted by GitBox <gi...@apache.org>.
SHuixo commented on issue #6330:
URL: https://github.com/apache/iceberg/issues/6330#issuecomment-1333486031
This situation is the same as I have encountered #6104, and in the current version, there is no effective way to support compressing historical data files containing delete operations during the data writing process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] commented on issue #6330: iceberg : format-version=2 , when the job is running (insert and update), can not execute rewrite small data file ?
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6330:
URL: https://github.com/apache/iceberg/issues/6330#issuecomment-1595902406
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] XBaith commented on issue #6330: iceberg : format-version=2 , when the job is running (insert and update), can not execute rewrite small data file ?
Posted by GitBox <gi...@apache.org>.
XBaith commented on issue #6330:
URL: https://github.com/apache/iceberg/issues/6330#issuecomment-1336311904
#4996
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] closed issue #6330: iceberg : format-version=2 , when the job is running (insert and update), can not execute rewrite small data file ?
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6330: iceberg : format-version=2 , when the job is running (insert and update), can not execute rewrite small data file ?
URL: https://github.com/apache/iceberg/issues/6330
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org