You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "puchengy (via GitHub)" <gi...@apache.org> on 2023/06/30 17:20:01 UTC
[GitHub] [iceberg] puchengy opened a new issue, #7949: Migrate/ snapshot action should exclude file that does not contain any record
puchengy opened a new issue, #7949:
URL: https://github.com/apache/iceberg/issues/7949
### Apache Iceberg version
1.3.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
The current behavior is migrate/ snapshot action will add the file even without any record into the Iceberg table. This part is fine but it create issues later on. For example, if an Iceberg table contains a partition that only have files without any record, `delete from table where partition = emptey_partition` will not be able to drop that partition.
Ideally we should skip adding those files to the table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zinking commented on issue #7949: Migrate/ snapshot action should exclude file that does not contain any record
Posted by "zinking (via GitHub)" <gi...@apache.org>.
zinking commented on issue #7949:
URL: https://github.com/apache/iceberg/issues/7949#issuecomment-1635124150
guess the question is still the same, if the file is empty with records, why is it included in the commit in the first place? isn't that supposed to be fixed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] puchengy commented on issue #7949: Migrate/ snapshot action should exclude file that does not contain any record
Posted by "puchengy (via GitHub)" <gi...@apache.org>.
puchengy commented on issue #7949:
URL: https://github.com/apache/iceberg/issues/7949#issuecomment-1635208101
@zinking https://github.com/apache/iceberg/pull/8040 is already doing this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zinking commented on issue #7949: Migrate/ snapshot action should exclude file that does not contain any record
Posted by "zinking (via GitHub)" <gi...@apache.org>.
zinking commented on issue #7949:
URL: https://github.com/apache/iceberg/issues/7949#issuecomment-1635177219
not really at the various engine side, but actually at the moment when it is included into iceberg. or to be precise during the `commit` process of iceberg, we can simply check and discard those empty files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] puchengy commented on issue #7949: Migrate/ snapshot action should exclude file that does not contain any record
Posted by "puchengy (via GitHub)" <gi...@apache.org>.
puchengy commented on issue #7949:
URL: https://github.com/apache/iceberg/issues/7949#issuecomment-1635128922
@zinking I see what you mean, it is hard to fix the source since these files are produced by various engines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org