You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Shekharrajak (via GitHub)" <gi...@apache.org> on 2025/02/16 12:51:56 UTC
[I] API to find out the number of datafiles deleted [iceberg]
Shekharrajak opened a new issue, #12288:
URL: https://github.com/apache/iceberg/issues/12288
### Query engine
JAVA API
### Question
Using JAVA API I would like to find out how many datafiles are deleted in my delete API and in current snapshot (comparing with previous snapshot).
Can anyone share how it is achievable ? Please review my code, and comment if it is a good way of doing the same, if there is no API:
```
Snapshot snapshot = table.snapshot(latestSnapshotId);
List<ManifestFile> manifestFiles = snapshot.allManifests(table.io());
for (ManifestFile manifest : manifestFiles) {
try (ManifestReader<DataFile> reader = ManifestFiles.read(manifest, table.io())) {
for (DataFile file : reader) {
if (manifest.hasDeletedFiles()) { // This ensures we're checking deleted files
LOGGER.info("Deleted DataFile: {}, Partition: {}, Manifest: {}",
file.path(), file.partition(), manifest.path());
}
}
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] API to find out the number of datafiles deleted [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #12288:
URL: https://github.com/apache/iceberg/issues/12288#issuecomment-2662410318
Hey @Shekharrajak, I think you're already pretty close, but I'm not sure I understand what you're looking for. The snapshot summary properties are described [here](https://iceberg.apache.org/spec/#optional-snapshot-summary-fields), maybe that helps :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] API to find out the number of datafiles deleted [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12288:
URL: https://github.com/apache/iceberg/issues/12288#issuecomment-3193995582
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] API to find out the number of datafiles deleted [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on issue #12288:
URL: https://github.com/apache/iceberg/issues/12288#issuecomment-2662370812
Here's a solution (using Spark SQL as an example).
1. Find added position delete files from the `entries` metadata table of a `snapshot_id`.
```sql
select data_file.file_path from csdw_t.F_CS_CNVRSTN_SEGMENT.entries where snapshot_id='$snapshot_id' and data_file.content=1;
```
2. Collect affected data files from content position delete files, whose columns are data_file_path and position.
```sql
select * from parquet.`$position_delete_file.parquet`
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] API to find out the number of datafiles deleted [iceberg]
Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on issue #12288:
URL: https://github.com/apache/iceberg/issues/12288#issuecomment-2662128376
I found a testcases :
```
Map<String, String> summary = table.currentSnapshot().summary();
assertThat(summary.get("deleted-data-files"))
.as("Deleted files count must match")
.isEqualTo("4");
```
But I want to understand if the whole datafile is marked as deleted or some data rows of the datafile marked as deleted - do we have other data rows also which is not deleted as part of deletion ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] API to find out the number of datafiles deleted [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #12288: API to find out the number of datafiles deleted
URL: https://github.com/apache/iceberg/issues/12288
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] API to find out the number of datafiles deleted [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on issue #12288:
URL: https://github.com/apache/iceberg/issues/12288#issuecomment-2661554039
If it's just the count you can use snapshot summary : https://iceberg.apache.org/spec/?h=spec#metrics
people also use partition summary to have partition level break down !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] API to find out the number of datafiles deleted [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12288:
URL: https://github.com/apache/iceberg/issues/12288#issuecomment-3240534491
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org