You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "naanagon (via GitHub)" <gi...@apache.org> on 2026/09/02 16:06:46 UTC
[PR] Skip empty files while migrating tables to iceberg [iceberg]
naanagon opened a new pull request, #17927:
URL: https://github.com/apache/iceberg/pull/17927
ORC:
```
java.lang.IllegalArgumentException: ORC schema does not contain Iceberg IDs
at org.apache.iceberg.orc.ORCSchemaUtil.convert(ORCSchemaUtil.java:293)
at org.apache.iceberg.orc.OrcMetrics.buildOrcMetrics(OrcMetrics.java:143)
at org.apache.iceberg.orc.OrcMetrics.fromInputFile(OrcMetrics.java:93)
at org.apache.iceberg.orc.OrcMetrics.fromInputFile(OrcMetrics.java:87)
at org.apache.iceberg.data.TableMigrationUtil.getOrcMetrics(TableMigrationUtil.java:296)
at org.apache.iceberg.data.TableMigrationUtil.lambda$listPartition$4(TableMigrationUtil.java:244)
```
Parquet:
```
java.lang.RuntimeException: mig_parquet_sample/date_key=2026-09-02/1_folder is not a Parquet file (length is too low: 0)
at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:599)
at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:578)
at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:971)
at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:961)
at org.apache.iceberg.shaded.org.apache.parquet.hadoop.ParquetFileReader.open(ParquetFileReader.java:718)
at org.apache.iceberg.parquet.ParquetUtil.fileMetrics(ParquetUtil.java:64)
at org.apache.iceberg.data.TableMigrationUtil.getParquetMetrics(TableMigrationUtil.java:287)
at org.apache.iceberg.data.TableMigrationUtil.lambda$listPartition$3(TableMigrationUtil.java:236)
```
Avro is working fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Skip empty files while migrating tables to iceberg [iceberg]
Posted by "ebyhr (via GitHub)" <gi...@apache.org>.
ebyhr commented on code in PR #17927:
URL: https://github.com/apache/iceberg/pull/17927#discussion_r3930950983
##########
data/src/main/java/org/apache/iceberg/data/TableMigrationUtil.java:
##########
Review Comment:
Why not exclude empty files here? `FileStatus` can return the length without additional call, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Skip empty files while migrating tables to iceberg [iceberg]
Posted by "naanagon (via GitHub)" <gi...@apache.org>.
naanagon commented on PR #17927:
URL: https://github.com/apache/iceberg/pull/17927#issuecomment-5513252103
In a bulk migration then this becomes prerequisite, find all the empty files for all tables and removing them, instead if migrate itself can handle it will be helpful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Data: Skip empty files while migrating tables to iceberg [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17927:
URL: https://github.com/apache/iceberg/pull/17927#issuecomment-6135310868
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Data: Skip empty files while migrating tables to iceberg [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #17927: Data: Skip empty files while migrating tables to iceberg
URL: https://github.com/apache/iceberg/pull/17927
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Skip empty files while migrating tables to iceberg [iceberg]
Posted by "ebyhr (via GitHub)" <gi...@apache.org>.
ebyhr commented on code in PR #17927:
URL: https://github.com/apache/iceberg/pull/17927#discussion_r3930970011
##########
data/src/test/java/org/apache/iceberg/data/TestTableMigrationUtil.java:
##########
@@ -94,6 +94,21 @@ void testListPartitionMissingFilesFailure() {
.hasRootCauseInstanceOf(FileNotFoundException.class);
}
+ @Test
+ void testListPartitionSkipsEmptyFiles() throws IOException {
Review Comment:
> Avro is working fine.
Is it ensured by the existing test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Skip empty files while migrating tables to iceberg [iceberg]
Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on PR #17927:
URL: https://github.com/apache/iceberg/pull/17927#issuecomment-5514634130
Suggest to rename the title as `Data: Skip empty files while migrating tables to iceberg` as current PR title check CI is failing.
Also, I believe Russell asked more about the context on why empty file exists In the first place per https://github.com/apache/iceberg/issues/17926#issuecomment-5512788636. Better to understand the problem first, if empty file problem is not recurring and can be handled by one time clean up before migrate to iceberg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Skip empty files while migrating tables to iceberg [iceberg]
Posted by "naanagon (via GitHub)" <gi...@apache.org>.
naanagon commented on code in PR #17927:
URL: https://github.com/apache/iceberg/pull/17927#discussion_r3947735210
##########
data/src/main/java/org/apache/iceberg/data/TableMigrationUtil.java:
##########
Review Comment:
yes makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Data: Skip empty files while migrating tables to iceberg [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17927:
URL: https://github.com/apache/iceberg/pull/17927#issuecomment-6304865912
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org