You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "hemanthboyina (via GitHub)" <gi...@apache.org> on 2025/11/26 17:09:05 UTC

[PR] rewrite_data_files with rewrite-all=true producting duplicate rows [iceberg]

hemanthboyina opened a new pull request, #14696:
URL: https://github.com/apache/iceberg/pull/14696

   The rewrite-all=true option incorrectly bypassed file group validation (filterFileGroups()), allowing single-file groups to be rewritten. This caused the commit phase to incorrectly handle file replacement, resulting in both old and new files being retained in the table metadata, leading to duplicate rows
   
   Fixes https://github.com/apache/iceberg/issues/14667


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] rewrite_data_files with rewrite-all=true producting duplicate rows [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #14696:
URL: https://github.com/apache/iceberg/pull/14696#issuecomment-3584885484

   The goal of the `rewrite-all=true` is to rewrite data files regardless of any filter. This is a useful feature as it rewrites the file with the Iceberg writer.
   
   We still need to fix the duplication, so I would like to understand how it could lead to data duplication. Could you please elaborate a bit on this:
   > This caused the commit phase to incorrectly handle file replacement, resulting in both old and new files being retained in the table metadata, leading to duplicate rows
   
   Thanks,
   Peter


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] rewrite_data_files with rewrite-all=true producting duplicate rows [iceberg]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on PR #14696:
URL: https://github.com/apache/iceberg/pull/14696#issuecomment-3606065055

   I am unable to reproduce the scenario in latest iceberg . Thanks for the clarification @pvary 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] rewrite_data_files with rewrite-all=true producting duplicate rows [iceberg]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina closed pull request #14696: rewrite_data_files with rewrite-all=true producting duplicate rows
URL: https://github.com/apache/iceberg/pull/14696


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] rewrite_data_files with rewrite-all=true producting duplicate rows [iceberg]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on PR #14696:
URL: https://github.com/apache/iceberg/pull/14696#issuecomment-3584239832

   the failed tests are because with the new fix we always apply filterFileGroups(), tests that use REWRITE_ALL=true are now getting empty results because the file groups don't meet the minimum criteria (like min-input-files=5 by default).
   
   @pvary could you please help in reviewing this change and suggest how to proceed here further.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org