You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/02/08 06:42:58 UTC
[GitHub] [iceberg] naushadahamad786 opened a new issue #4063: 'total-delete-files' metadata json attribute value is always '0', Even we apply 'Merge with Update' sql query
naushadahamad786 opened a new issue #4063:
URL: https://github.com/apache/iceberg/issues/4063
-SQL Query
MERGE INTO local.db.Mergetable1 t USING (SELECT * FROM local.db.Mergetable2) u ON t.id = u.id WHEN MATCHED AND t.id=1 THEN UPDATE SET t.data='z'
We are expecting '"total-delete-files" attribute value should be 1, but it's show '0'.
"summary" : {
"operation" : "overwrite",
"spark.app.id" : "local-1644210838923",
"deleted-data-files" : "1",
"deleted-records" : "2",
"removed-files-size" : "658",
"changed-partition-count" : "1",
"total-records" : "0",
"total-files-size" : "0",
"total-data-files" : "0",
"total-delete-files" : "0",
"total-position-deletes" : "0",
"total-equality-deletes" : "0"
}
Please find attached file for 'metadata.json' file.
[v7.metadata.zip](https://github.com/apache/iceberg/files/8021379/v7.metadata.zip)
If I missed any configuration to reproduce the "total-delete-files" , please help me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #4063: Delete files are not getting generate by using format-version 1 and 2
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #4063:
URL: https://github.com/apache/iceberg/issues/4063#issuecomment-1037256324
Did you set ```TableProperties.UPDATE_MODE, "merge-on-read"```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] naushadahamad786 commented on issue #4063: Delete files are not getting generate by using format-version 1 and 2
Posted by GitBox <gi...@apache.org>.
naushadahamad786 commented on issue #4063:
URL: https://github.com/apache/iceberg/issues/4063#issuecomment-1036219406
I have used also below version but We are not able to find any delete files-
Spark Version 3.2.0
Iceberg Version 0.13.1
and
iceberg-spark-runtime-3.2_2.12
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer edited a comment on issue #4063: Delete files are not getting generate by using format-version 1 and 2
Posted by GitBox <gi...@apache.org>.
RussellSpitzer edited a comment on issue #4063:
URL: https://github.com/apache/iceberg/issues/4063#issuecomment-1037256324
Did you set ```TableProperties.UPDATE_MODE, "merge-on-read"```
https://github.com/apache/iceberg/blob/master/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMergeOnReadUpdate.java#L33-L38
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zhongyujiang commented on issue #4063: Delete files are not getting generate by using format-version 1 and 2
Posted by GitBox <gi...@apache.org>.
zhongyujiang commented on issue #4063:
URL: https://github.com/apache/iceberg/issues/4063#issuecomment-1037021370
@akghbti @naushadahamad786 0.13 supports merge-on-read delete: [0.13.0 Release Notes](https://iceberg.apache.org/releases/), and merge-on-read merge is still in progress, you can track progress [here](https://github.com/apache/iceberg/projects/11).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] akghbti edited a comment on issue #4063: Delete files are not getting generate by using format-version 1 and 2
Posted by GitBox <gi...@apache.org>.
akghbti edited a comment on issue #4063:
URL: https://github.com/apache/iceberg/issues/4063#issuecomment-1034016407
Even 0.13 is not producing delete files with Spark 3.2 . Any config needs to be set for the same ??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #4063: Delete files are not getting generate by using format-version 1 and 2
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #4063:
URL: https://github.com/apache/iceberg/issues/4063#issuecomment-1037256324
Did you set ```TableProperties.UPDATE_MODE, "merge-on-read"```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer edited a comment on issue #4063: Delete files are not getting generate by using format-version 1 and 2
Posted by GitBox <gi...@apache.org>.
RussellSpitzer edited a comment on issue #4063:
URL: https://github.com/apache/iceberg/issues/4063#issuecomment-1037256324
Did you set ```TableProperties.UPDATE_MODE, "merge-on-read"```
https://github.com/apache/iceberg/blob/master/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMergeOnReadUpdate.java#L33-L38
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] akghbti commented on issue #4063: Delete files are not getting generate by using format-version 1 and 2
Posted by GitBox <gi...@apache.org>.
akghbti commented on issue #4063:
URL: https://github.com/apache/iceberg/issues/4063#issuecomment-1034016407
Even 0.13 is not producing delete files with Spar 3.2 . Any config needs to be set for the same ??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zhongyujiang commented on issue #4063: Delete files are not getting generate by using format-version 1 and 2
Posted by GitBox <gi...@apache.org>.
zhongyujiang commented on issue #4063:
URL: https://github.com/apache/iceberg/issues/4063#issuecomment-1032554824
@naushadahamad786 0.12.1 only supports copy-on-write MERGE INTO, matching data files will be replaced with new data files immediately, so it won't produce delete files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org