You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/05/01 11:51:11 UTC
[GitHub] [iceberg] chenjunjiedada opened a new pull request, #4677: Core: add the file count of specific deletes in the snapshot summary
chenjunjiedada opened a new pull request, #4677:
URL: https://github.com/apache/iceberg/pull/4677
This adds more info for deletes in the snapshot summary.
In the case of V2 tables, users may have offline service to compact deletes. For example, in our production, we analyze the snapshot summary from the commit event to determine whether to start major compaction or minor compaction. The detailed summary info about deletes can help us calculate better compaction timing and resource.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] singhpk234 commented on a diff in pull request #4677: Core: add the file count of specific deletes in the snapshot summary
Posted by GitBox <gi...@apache.org>.
singhpk234 commented on code in PR #4677:
URL: https://github.com/apache/iceberg/pull/4677#discussion_r862600196
##########
core/src/main/java/org/apache/iceberg/SnapshotSummary.java:
##########
@@ -236,6 +248,10 @@ void clear() {
void addTo(ImmutableMap.Builder<String, String> builder) {
setIf(addedFiles > 0, builder, ADDED_FILES_PROP, addedFiles);
setIf(removedFiles > 0, builder, DELETED_FILES_PROP, removedFiles);
+ setIf(addedEqDeleteFiles > 0, builder, ADD_EQ_DELETE_FILES_PROP, addedEqDeleteFiles);
+ setIf(removedEqDeleteFiles > 0, builder, REMOVED_EQ_DELETE_FILES_PROP, removedEqDeleteFiles);
+ setIf(addedPosDeleteFiles > 0, builder, ADD_POS_DELETE_FILES_PROP, addedPosDeleteFiles);
+ setIf(removedPosDeleteFiles > 0, builder, REMOVED_POS_DELETE_FILES_PROP, removedPosDeleteFiles);
setIf(addedDeleteFiles > 0, builder, ADDED_DELETE_FILES_PROP, addedDeleteFiles);
setIf(removedDeleteFiles > 0, builder, REMOVED_DELETE_FILES_PROP, removedDeleteFiles);
Review Comment:
[question] since now we are tracking added / removed delete files at a more granular level (i.e pos delete added / removed & eq delete added / removed delete files) these metrics can be re-created using them, should we remove or deprecate them from snapshot summary ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4677: Core: add the file count of specific deletes in the snapshot summary
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4677:
URL: https://github.com/apache/iceberg/pull/4677#discussion_r863107149
##########
core/src/main/java/org/apache/iceberg/SnapshotSummary.java:
##########
@@ -236,6 +248,10 @@ void clear() {
void addTo(ImmutableMap.Builder<String, String> builder) {
setIf(addedFiles > 0, builder, ADDED_FILES_PROP, addedFiles);
setIf(removedFiles > 0, builder, DELETED_FILES_PROP, removedFiles);
+ setIf(addedEqDeleteFiles > 0, builder, ADD_EQ_DELETE_FILES_PROP, addedEqDeleteFiles);
+ setIf(removedEqDeleteFiles > 0, builder, REMOVED_EQ_DELETE_FILES_PROP, removedEqDeleteFiles);
+ setIf(addedPosDeleteFiles > 0, builder, ADD_POS_DELETE_FILES_PROP, addedPosDeleteFiles);
+ setIf(removedPosDeleteFiles > 0, builder, REMOVED_POS_DELETE_FILES_PROP, removedPosDeleteFiles);
setIf(addedDeleteFiles > 0, builder, ADDED_DELETE_FILES_PROP, addedDeleteFiles);
setIf(removedDeleteFiles > 0, builder, REMOVED_DELETE_FILES_PROP, removedDeleteFiles);
Review Comment:
Deprecating things is quite a bit of work for little gain here. I'd just continue to update them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #4677: Core: add the file count of specific deletes in the snapshot summary
Posted by GitBox <gi...@apache.org>.
rdblue merged PR #4677:
URL: https://github.com/apache/iceberg/pull/4677
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #4677: Core: add the file count of specific deletes in the snapshot summary
Posted by GitBox <gi...@apache.org>.
rdblue commented on PR #4677:
URL: https://github.com/apache/iceberg/pull/4677#issuecomment-1115257440
Thanks, @chenjunjiedada!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org