You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "damianm-dev (via GitHub)" <gi...@apache.org> on 2025/10/16 10:09:02 UTC
[I] rewriting data files within a multi-level partition structure [iceberg]
damianm-dev opened a new issue, #14345:
URL: https://github.com/apache/iceberg/issues/14345
### Query engine
Spark 3.5.6
### Question
Does Apache Iceberg 1.10 support rewriting data files within a multi-level partition structure such as month_key=10-2025/date_key=14102025?
And is it sufficient to use the min-input-files option — as shown in the documentation
CALL catalog_name.system.rewrite_data_files(
table => 'db.sample',
options => map('min-input-files','2')
);
to automatically compact small files in such partitions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] rewriting data files within a multi-level partition structure [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14345:
URL: https://github.com/apache/iceberg/issues/14345#issuecomment-4277198005
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] rewriting data files within a multi-level partition structure [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14345:
URL: https://github.com/apache/iceberg/issues/14345#issuecomment-4367589382
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] rewriting data files within a multi-level partition structure [iceberg]
Posted by "peach12345 (via GitHub)" <gi...@apache.org>.
peach12345 commented on issue #14345:
URL: https://github.com/apache/iceberg/issues/14345#issuecomment-3424890185
Hi @damianm-dev
We are currently using the Iceberg 1.10 rewrite job. It appears that the job supports multi-level partition structures and compacts Parquet files within those folders/partitions.
To delete outdated Parquet files, we use the deleteOrphanFiles job.
We are currently using the following settings:
scheduleOnInterval(10 minutes)
minInputFiles(2)
maxFileGroupSizeBytes(1GB)
For deleteOrphanFiles:
deleteBatchSize(100)
scheduleOnInterval(10 mins)
minAge(1 hour)
For ExpireSnapshots:
deleteBatchSize(100)
scheduleOnInterval(10 mins)
maxSnapshotAge(1 hour)
We tried running the rewrite job without setting minInputFiles, but in that case the job never completed and threw exceptions like: "buffer pool is already destroyed."
However, the ExpireSnapshots job doesn't seem to be working as expected — it’s not deleting old snapshot metadata files. If anyone has experience with the ExpireSnapshots job or has encountered similar issues, we’d appreciate any insights! :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] rewriting data files within a multi-level partition structure [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #14345: rewriting data files within a multi-level partition structure
URL: https://github.com/apache/iceberg/issues/14345
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org