You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "laskoviymishka (via GitHub)" <gi...@apache.org> on 2026/04/14 08:41:29 UTC
[PR] fix(table): warn on row count mismatch in Overwrite [iceberg-go]
laskoviymishka opened a new pull request, #894:
URL: https://github.com/apache/iceberg-go/pull/894
Add a diagnostic slog.Warn in Transaction.Overwrite when the newly written data files contain fewer rows than the files being deleted. This surfaces silent row loss from broken RecordReader adapters or iterator issues that would otherwise go unnoticed.
The warning is non-blocking — callers that intentionally reduce data via overwrite are not affected. Ref #860.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): warn on row count mismatch in Overwrite [iceberg-go]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #894:
URL: https://github.com/apache/iceberg-go/pull/894#discussion_r3089658715
##########
table/transaction.go:
##########
@@ -1057,6 +1058,24 @@ func (t *Transaction) Overwrite(ctx context.Context, rdr array.RecordReader, sna
updater.appendDataFile(df)
}
+ // Diagnostic: warn when an overwrite writes fewer rows than it deletes.
+ // This is not always a bug (the caller may intentionally reduce data),
+ // but it surfaces silent row loss from broken RecordReader adapters
+ // or iterator issues that would otherwise go unnoticed (#860).
+ var addedRows, deletedRows int64
+ for _, df := range updater.addedFiles {
+ addedRows += df.Count()
+ }
+ for _, df := range updater.deletedFiles {
+ deletedRows += df.Count()
+ }
+ if deletedRows > 0 && addedRows < deletedRows {
+ slog.Warn("Overwrite produced fewer rows than deleted",
+ "added_rows", addedRows,
+ "deleted_rows", deletedRows,
+ "delta", addedRows-deletedRows)
+ }
Review Comment:
yes, this actually a good idea, logs in such systems usually should be treated as part of behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): warn on row count mismatch in Overwrite [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #894:
URL: https://github.com/apache/iceberg-go/pull/894#discussion_r3082152735
##########
table/transaction.go:
##########
@@ -1057,6 +1058,24 @@ func (t *Transaction) Overwrite(ctx context.Context, rdr array.RecordReader, sna
updater.appendDataFile(df)
}
+ // Diagnostic: warn when an overwrite writes fewer rows than it deletes.
+ // This is not always a bug (the caller may intentionally reduce data),
+ // but it surfaces silent row loss from broken RecordReader adapters
+ // or iterator issues that would otherwise go unnoticed (#860).
+ var addedRows, deletedRows int64
+ for _, df := range updater.addedFiles {
+ addedRows += df.Count()
+ }
+ for _, df := range updater.deletedFiles {
+ deletedRows += df.Count()
+ }
+ if deletedRows > 0 && addedRows < deletedRows {
+ slog.Warn("Overwrite produced fewer rows than deleted",
+ "added_rows", addedRows,
+ "deleted_rows", deletedRows,
+ "delta", addedRows-deletedRows)
+ }
Review Comment:
should we add a test for this? We can use an slog adapter so that we can capture the slog output etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): warn on row count mismatch in Overwrite [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade merged PR #894:
URL: https://github.com/apache/iceberg-go/pull/894
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org