You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "slfan1989 (via GitHub)" <gi...@apache.org> on 2025/08/16 05:33:37 UTC
[PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
slfan1989 opened a new pull request, #13831:
URL: https://github.com/apache/iceberg/pull/13831
## Summary:
This pull request enhances the DeleteFilesProcessor by adding time metrics for file deletion operations. Additionally, the unit tests for DeleteFilesProcessor are extended with several edge cases and failure scenarios, ensuring the processor works correctly under various conditions, including file deletion failures, large files, batch deletions, and concurrent deletions.
## Main Changes:
### Added Time Metrics:
Introduced time metrics to DeleteFilesProcessor to track the time taken for each file deletion operation, helping monitor the performance of delete operations.
The Histogram is used to measure and log the time taken for file deletion operations, providing detailed performance data.
### Extended Unit Tests:
- Delete Non-Existent File: Tests the scenario where a file does not exist, ensuring the system handles this case without throwing exceptions.
- Delete Large File: Simulates the deletion of a large file, ensuring the processor can handle large file deletions correctly and performs well.
- Batch Delete Boundary Test: Tests batch file deletion to ensure files are deleted in batches once the batch size is reached.
- Concurrent Delete: Simulates concurrent file deletions, ensuring stability when handling concurrent delete operations.
- Delete With Failure: Simulates file deletion failures, ensuring the processor can handle failures and correctly record the number of failed deletions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#discussion_r2282574969
##########
flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/DeleteFilesProcessor.java:
##########
@@ -76,6 +79,10 @@ public void open() throws Exception {
taskMetricGroup.counter(TableMaintenanceMetrics.DELETE_FILE_FAILED_COUNTER);
this.succeededCounter =
taskMetricGroup.counter(TableMaintenanceMetrics.DELETE_FILE_SUCCEEDED_COUNTER);
+ this.deleteFileTimeMsHistogram =
+ taskMetricGroup.histogram(
+ TableMaintenanceMetrics.DELETE_FILE_TIME_MS_HISTOGRAM,
+ new DescriptiveStatisticsHistogram(1000));
Review Comment:
@mxm, @Guosmilesmile: Is this a useful metrics based on your experience too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #13831: Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests.
URL: https://github.com/apache/iceberg/pull/13831
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#discussion_r2314316256
##########
flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestDeleteFilesProcessor.java:
##########
@@ -74,27 +75,158 @@ void testDeleteMissingFile() throws Exception {
Path dummyFile =
FileSystems.getDefault().getPath(table.location().substring(5), DUMMY_FILE_NAME);
- deleteFile(tableLoader(), dummyFile.toString());
+ deleteFile(tableLoader(), dummyFile.toString(), true /* expectSuccess */);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
@Test
void testInvalidURIScheme() throws Exception {
- deleteFile(tableLoader(), "wrong://");
+ deleteFile(tableLoader(), "wrong://", false /* expectFail */);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
- private void deleteFile(TableLoader tableLoader, String fileName) throws Exception {
- tableLoader().open();
+ @Test
+ void testDeleteNonExistentFile() throws Exception {
+ String nonexistentFile = "nonexistentFile.txt";
+
+ deleteFile(tableLoader(), nonexistentFile, true /* expectSuccess */);
+
+ assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
+ }
+
+ @Test
+ void testDelete10MBFile() throws Exception {
+ // Simulate a large file (e.g., 10MB file)
+ String largeFileName = "largeFile.txt";
+ Path largeFile = Path.of(tablePath(table).toString(), largeFileName);
+
+ // Write a large file to disk (this will simulate the large file in the filesystem)
+ byte[] largeData = new byte[1024 * 1024 * 10]; // 10 MB
+ Files.write(largeFile, largeData);
+
+ // Verify that the file was created
+ Set<String> files = listFiles(table);
+ assertThat(files).contains(largeFileName);
+
+ // Use the DeleteFilesProcessor to delete the large file
+ deleteFile(tableLoader(), largeFile.toString(), true /* expectSuccess */);
+
+ // Verify that the large file has been deleted
+ files = listFiles(table);
+ assertThat(files).doesNotContain(largeFileName);
+ }
Review Comment:
I'm sure that we can delete a 10Mb file, at least on local file system. I'm not sure what we are testing here. Why does the size matter here?
##########
flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/DeleteFilesProcessor.java:
##########
@@ -76,6 +79,10 @@ public void open() throws Exception {
taskMetricGroup.counter(TableMaintenanceMetrics.DELETE_FILE_FAILED_COUNTER);
this.succeededCounter =
taskMetricGroup.counter(TableMaintenanceMetrics.DELETE_FILE_SUCCEEDED_COUNTER);
+ this.deleteFileTimeMsHistogram =
+ taskMetricGroup.histogram(
+ TableMaintenanceMetrics.DELETE_FILE_TIME_MS_HISTOGRAM,
+ new DescriptiveStatisticsHistogram(1000));
Review Comment:
We haven't required the need to measure delete file duration. We are generally more concerned with the total time the maintenance takes. I would assume that the actual deletion of the files is a large chunk, but we have managed storage, so there is usually very little we can do about it. Maybe it would suffice to keep the max duration here instead of a full histogram?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#discussion_r2312489039
##########
flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestDeleteFilesProcessor.java:
##########
@@ -74,27 +79,183 @@ void testDeleteMissingFile() throws Exception {
Path dummyFile =
FileSystems.getDefault().getPath(table.location().substring(5), DUMMY_FILE_NAME);
- deleteFile(tableLoader(), dummyFile.toString());
+ deleteFile(tableLoader(), dummyFile.toString(), true /* expectSuccess */);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
@Test
void testInvalidURIScheme() throws Exception {
- deleteFile(tableLoader(), "wrong://");
+ deleteFile(tableLoader(), "wrong://", false /* expectFail */);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
- private void deleteFile(TableLoader tableLoader, String fileName) throws Exception {
- tableLoader().open();
+ @Test
+ void testDeleteNonExistentFile() throws Exception {
+ String nonexistentFile = "nonexistentFile.txt";
+
+ deleteFile(tableLoader(), nonexistentFile, true /* expectSuccess */);
+
+ assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
+ }
+
+ @Test
+ void testDelete10MBFile() throws Exception {
+ // Simulate a large file (e.g., 10MB file)
+ String largeFileName = "largeFile.txt";
+ Path largeFile = Path.of(tablePath(table).toString(), largeFileName);
+
+ // Write a large file to disk (this will simulate the large file in the filesystem)
+ byte[] largeData = new byte[1024 * 1024 * 10]; // 10 MB
+ Files.write(largeFile, largeData);
+
+ // Verify that the file was created
+ Set<String> files = listFiles(table);
+ assertThat(files).contains(largeFileName);
+
+ // Use the DeleteFilesProcessor to delete the large file
+ deleteFile(tableLoader(), largeFile.toString(), true /* expectSuccess */);
+
+ // Verify that the large file has been deleted
+ files = listFiles(table);
+ assertThat(files).doesNotContain(largeFileName);
+ }
+
+ @Test
+ void testBatchDelete() throws Exception {
+ // Simulate adding multiple files
+ Set<String> filesToDelete = Sets.newHashSet(TABLE_FILES);
+ filesToDelete.add("file1.txt");
+ filesToDelete.add("file2.txt");
+
+ // Use a smaller batch size to trigger batch deletion logic
+ DeleteFilesProcessor deleteFilesProcessor =
+ new DeleteFilesProcessor(table, DUMMY_TASK_NAME, 0, 2);
try (OneInputStreamOperatorTestHarness<String, Void> testHarness =
- new OneInputStreamOperatorTestHarness<>(
- new DeleteFilesProcessor(table, DUMMY_TASK_NAME, 0, 10), StringSerializer.INSTANCE)) {
+ new OneInputStreamOperatorTestHarness<>(deleteFilesProcessor, StringSerializer.INSTANCE)) {
testHarness.open();
- testHarness.processElement(fileName, System.currentTimeMillis());
+
+ for (String file : filesToDelete) {
+ testHarness.processElement(file, System.currentTimeMillis());
+ }
+
+ testHarness.processWatermark(EVENT_TIME);
+ testHarness.endInput();
+
+ // Verify that files are deleted
+ assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
+ assertThat(deleteFilesProcessor.getSucceededCounter().getCount())
+ .isEqualTo(filesToDelete.size());
+ assertThat(deleteFilesProcessor.getFailedCounter().getCount()).isEqualTo(0);
+ assertThat(deleteFilesProcessor.getDeleteFileTimeMsHistogram().getStatistics().getMean())
+ .isGreaterThan(0);
+ } finally {
+ deleteFilesProcessor.close();
+ }
+ }
+
+ @Test
+ void testConcurrentDelete() throws Exception {
+ Path root = tablePath(table);
+
+ // Generate 30 test files: delete-0.txt ... delete-29.txt
+ Set<String> targets = Sets.newHashSet();
+ for (int i = 0; i < 30; i++) {
+ targets.add("delete-" + i + ".txt");
+ }
+
+ for (String f : targets) {
+ Files.write(root.resolve(f), f.getBytes(StandardCharsets.UTF_8));
+ }
Review Comment:
I have made improvements to this part of the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#issuecomment-3242765238
@pvary Could you please take a look at this PR? Thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#discussion_r2282596363
##########
flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestDeleteFilesProcessor.java:
##########
@@ -74,27 +79,172 @@ void testDeleteMissingFile() throws Exception {
Path dummyFile =
FileSystems.getDefault().getPath(table.location().substring(5), DUMMY_FILE_NAME);
- deleteFile(tableLoader(), dummyFile.toString());
+ deleteFile(tableLoader(), dummyFile.toString(), true);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
@Test
void testInvalidURIScheme() throws Exception {
- deleteFile(tableLoader(), "wrong://");
+ deleteFile(tableLoader(), "wrong://", false);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
- private void deleteFile(TableLoader tableLoader, String fileName) throws Exception {
- tableLoader().open();
+ @Test
+ void testDeleteNonExistentFile() throws Exception {
+ String nonexistentFile = "nonexistentFile.txt";
+
+ deleteFile(tableLoader(), nonexistentFile, true);
+
+ assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
+ }
+
+ @Test
+ void testDeleteLargeFile() throws Exception {
+ // Simulate a large file (e.g., 100MB file)
+ String largeFileName = "largeFile.txt";
+ Path largeFile = Path.of(tablePath(table).toString(), largeFileName);
+
+ // Write a large file to disk (this will simulate the large file in the filesystem)
+ byte[] largeData = new byte[1024 * 1024 * 100]; // 100 MB
+ Files.write(largeFile, largeData);
+
+ // Verify that the file was created
+ Set<String> files = listFiles(table);
+ assertThat(files).contains(largeFileName);
+
+ // Use the DeleteFilesProcessor to delete the large file
+ deleteFile(tableLoader(), largeFile.toString(), true);
+
+ // Verify that the large file has been deleted
+ files = listFiles(table);
+ assertThat(files).doesNotContain(largeFileName);
+ }
+
+ private void deleteFile(TableLoader tableLoader, String fileName, boolean expectSuccess)
+ throws Exception {
+ tableLoader.open();
+ DeleteFilesProcessor deleteFilesProcessor =
+ new DeleteFilesProcessor(table, DUMMY_TASK_NAME, 0, 10);
try (OneInputStreamOperatorTestHarness<String, Void> testHarness =
- new OneInputStreamOperatorTestHarness<>(
- new DeleteFilesProcessor(table, DUMMY_TASK_NAME, 0, 10), StringSerializer.INSTANCE)) {
+ new OneInputStreamOperatorTestHarness<>(deleteFilesProcessor, StringSerializer.INSTANCE)) {
testHarness.open();
testHarness.processElement(fileName, System.currentTimeMillis());
testHarness.processWatermark(EVENT_TIME);
testHarness.endInput();
+
+ // Validate if the metrics meet expectations
+ if (expectSuccess) {
+ assertThat(deleteFilesProcessor.getSucceededCounter().getCount()).isEqualTo(1);
+ assertThat(deleteFilesProcessor.getFailedCounter().getCount()).isEqualTo(0);
+ assertThat(deleteFilesProcessor.getDeleteFileTimeMsHistogram().getStatistics().getMean())
+ .isGreaterThan(0);
+ } else {
+ assertThat(deleteFilesProcessor.getSucceededCounter().getCount()).isEqualTo(0);
+ assertThat(deleteFilesProcessor.getFailedCounter().getCount()).isEqualTo(1);
+ assertThat(deleteFilesProcessor.getDeleteFileTimeMsHistogram().getStatistics().getMean())
+ .isGreaterThan(0);
+ }
+
+ } finally {
+ deleteFilesProcessor.close();
+ }
+ }
+
+ @Test
+ void testBatchDelete() throws Exception {
+ // Simulate adding multiple files
+ Set<String> filesToDelete = Sets.newHashSet(TABLE_FILES);
+ filesToDelete.add("file1.txt");
+ filesToDelete.add("file2.txt");
+
+ // Use a smaller batch size to trigger batch deletion logic
+ DeleteFilesProcessor deleteFilesProcessor =
+ new DeleteFilesProcessor(table, DUMMY_TASK_NAME, 0, 2);
+ try (OneInputStreamOperatorTestHarness<String, Void> testHarness =
+ new OneInputStreamOperatorTestHarness<>(deleteFilesProcessor, StringSerializer.INSTANCE)) {
+ testHarness.open();
+
+ for (String file : filesToDelete) {
+ testHarness.processElement(file, System.currentTimeMillis());
+ }
+
+ testHarness.processWatermark(EVENT_TIME);
+ testHarness.endInput();
+
+ // Verify that files are deleted
+ assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
+ assertThat(deleteFilesProcessor.getSucceededCounter().getCount())
+ .isEqualTo(filesToDelete.size());
+ assertThat(deleteFilesProcessor.getFailedCounter().getCount()).isEqualTo(0);
+ assertThat(deleteFilesProcessor.getDeleteFileTimeMsHistogram().getStatistics().getMean())
+ .isGreaterThan(0);
+ } finally {
+ deleteFilesProcessor.close();
+ }
+ }
+
+ @Test
+ void testConcurrentDelete() throws Exception {
Review Comment:
How often this delete actually happens concurrently?
Like one file deleted by the `DeleteFilesProcessor`, and one file deleted by the `executorService`?
If we want to test real concurrency, we likely need to use more files, and maybe some delays.
If we are ok with checking concurrently deleted files, then `testDeleteNonExistentFile` might be enough
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#discussion_r2287555101
##########
flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/DeleteFilesProcessor.java:
##########
@@ -76,6 +79,10 @@ public void open() throws Exception {
taskMetricGroup.counter(TableMaintenanceMetrics.DELETE_FILE_FAILED_COUNTER);
this.succeededCounter =
taskMetricGroup.counter(TableMaintenanceMetrics.DELETE_FILE_SUCCEEDED_COUNTER);
+ this.deleteFileTimeMsHistogram =
+ taskMetricGroup.histogram(
+ TableMaintenanceMetrics.DELETE_FILE_TIME_MS_HISTOGRAM,
+ new DescriptiveStatisticsHistogram(1000));
Review Comment:
I really appreciate your valuable input, @pvary. I’ll take some time to reflect on your suggestions and share a considered reply soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#discussion_r2312498153
##########
flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestDeleteFilesProcessor.java:
##########
@@ -74,27 +79,183 @@ void testDeleteMissingFile() throws Exception {
Path dummyFile =
FileSystems.getDefault().getPath(table.location().substring(5), DUMMY_FILE_NAME);
- deleteFile(tableLoader(), dummyFile.toString());
+ deleteFile(tableLoader(), dummyFile.toString(), true /* expectSuccess */);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
@Test
void testInvalidURIScheme() throws Exception {
- deleteFile(tableLoader(), "wrong://");
+ deleteFile(tableLoader(), "wrong://", false /* expectFail */);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
- private void deleteFile(TableLoader tableLoader, String fileName) throws Exception {
- tableLoader().open();
+ @Test
+ void testDeleteNonExistentFile() throws Exception {
+ String nonexistentFile = "nonexistentFile.txt";
+
+ deleteFile(tableLoader(), nonexistentFile, true /* expectSuccess */);
+
+ assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
+ }
+
+ @Test
+ void testDelete10MBFile() throws Exception {
+ // Simulate a large file (e.g., 10MB file)
+ String largeFileName = "largeFile.txt";
+ Path largeFile = Path.of(tablePath(table).toString(), largeFileName);
+
+ // Write a large file to disk (this will simulate the large file in the filesystem)
+ byte[] largeData = new byte[1024 * 1024 * 10]; // 10 MB
+ Files.write(largeFile, largeData);
+
+ // Verify that the file was created
+ Set<String> files = listFiles(table);
+ assertThat(files).contains(largeFileName);
+
+ // Use the DeleteFilesProcessor to delete the large file
+ deleteFile(tableLoader(), largeFile.toString(), true /* expectSuccess */);
+
+ // Verify that the large file has been deleted
+ files = listFiles(table);
+ assertThat(files).doesNotContain(largeFileName);
+ }
+
+ @Test
+ void testBatchDelete() throws Exception {
+ // Simulate adding multiple files
+ Set<String> filesToDelete = Sets.newHashSet(TABLE_FILES);
+ filesToDelete.add("file1.txt");
+ filesToDelete.add("file2.txt");
+
+ // Use a smaller batch size to trigger batch deletion logic
+ DeleteFilesProcessor deleteFilesProcessor =
+ new DeleteFilesProcessor(table, DUMMY_TASK_NAME, 0, 2);
try (OneInputStreamOperatorTestHarness<String, Void> testHarness =
- new OneInputStreamOperatorTestHarness<>(
- new DeleteFilesProcessor(table, DUMMY_TASK_NAME, 0, 10), StringSerializer.INSTANCE)) {
+ new OneInputStreamOperatorTestHarness<>(deleteFilesProcessor, StringSerializer.INSTANCE)) {
testHarness.open();
- testHarness.processElement(fileName, System.currentTimeMillis());
+
+ for (String file : filesToDelete) {
+ testHarness.processElement(file, System.currentTimeMillis());
+ }
+
+ testHarness.processWatermark(EVENT_TIME);
+ testHarness.endInput();
+
+ // Verify that files are deleted
+ assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
+ assertThat(deleteFilesProcessor.getSucceededCounter().getCount())
+ .isEqualTo(filesToDelete.size());
+ assertThat(deleteFilesProcessor.getFailedCounter().getCount()).isEqualTo(0);
+ assertThat(deleteFilesProcessor.getDeleteFileTimeMsHistogram().getStatistics().getMean())
+ .isGreaterThan(0);
+ } finally {
+ deleteFilesProcessor.close();
+ }
+ }
+
+ @Test
+ void testConcurrentDelete() throws Exception {
+ Path root = tablePath(table);
+
+ // Generate 30 test files: delete-0.txt ... delete-29.txt
+ Set<String> targets = Sets.newHashSet();
+ for (int i = 0; i < 30; i++) {
+ targets.add("delete-" + i + ".txt");
+ }
+
+ for (String f : targets) {
+ Files.write(root.resolve(f), f.getBytes(StandardCharsets.UTF_8));
+ }
+ assertThat(listFiles(table)).containsAll(targets);
+
+ DeleteFilesProcessor p1 = new DeleteFilesProcessor(table, DUMMY_TASK_NAME + "-p1", 0, 2);
+ DeleteFilesProcessor p2 = new DeleteFilesProcessor(table, DUMMY_TASK_NAME + "-p2", 0, 2);
+
+ // Two processors that will try to delete the same files concurrently
+ try (OneInputStreamOperatorTestHarness<String, Void> h1 =
+ new OneInputStreamOperatorTestHarness<>(p1, StringSerializer.INSTANCE);
+ OneInputStreamOperatorTestHarness<String, Void> h2 =
+ new OneInputStreamOperatorTestHarness<>(p2, StringSerializer.INSTANCE)) {
+ h1.open();
+ h2.open();
+
+ // One barrier per file: ensures p1 and p2 try to delete the same file at the same time
+ Map<String, CyclicBarrier> barriers = Maps.newHashMap();
+ targets.forEach(f -> barriers.put(f, new CyclicBarrier(2)));
+
+ long ts = System.currentTimeMillis();
+
+ Thread t1 =
+ new Thread(
+ () -> {
+ try {
+ for (String f : targets) {
+ barriers.get(f).await(2, TimeUnit.SECONDS);
+ h1.processElement(f, ts);
+ }
+ h1.processWatermark(EVENT_TIME);
+ h1.endInput();
+ } catch (Exception ignored) {
+ }
+ },
+ "deleter-p1");
Review Comment:
Thank you for your question; it's a great one. The purpose of designing this unit test was to simulate a scenario where two Flink tasks are concurrently performing expiration snapshot cleanup, which leads to the concurrent deletion of metadata files. To model this, I used threads. After carefully considering your suggestion, I realized that using a loop to sequentially process both processors for deletion is also a valid approach. I've since made the necessary improvements to the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#discussion_r2282584449
##########
flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestDeleteFilesProcessor.java:
##########
@@ -74,27 +79,172 @@ void testDeleteMissingFile() throws Exception {
Path dummyFile =
FileSystems.getDefault().getPath(table.location().substring(5), DUMMY_FILE_NAME);
- deleteFile(tableLoader(), dummyFile.toString());
+ deleteFile(tableLoader(), dummyFile.toString(), true);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
@Test
void testInvalidURIScheme() throws Exception {
- deleteFile(tableLoader(), "wrong://");
+ deleteFile(tableLoader(), "wrong://", false);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
- private void deleteFile(TableLoader tableLoader, String fileName) throws Exception {
- tableLoader().open();
+ @Test
+ void testDeleteNonExistentFile() throws Exception {
+ String nonexistentFile = "nonexistentFile.txt";
+
+ deleteFile(tableLoader(), nonexistentFile, true);
+
+ assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
+ }
+
+ @Test
+ void testDeleteLargeFile() throws Exception {
+ // Simulate a large file (e.g., 100MB file)
+ String largeFileName = "largeFile.txt";
+ Path largeFile = Path.of(tablePath(table).toString(), largeFileName);
+
+ // Write a large file to disk (this will simulate the large file in the filesystem)
+ byte[] largeData = new byte[1024 * 1024 * 100]; // 100 MB
+ Files.write(largeFile, largeData);
+
+ // Verify that the file was created
+ Set<String> files = listFiles(table);
+ assertThat(files).contains(largeFileName);
+
+ // Use the DeleteFilesProcessor to delete the large file
+ deleteFile(tableLoader(), largeFile.toString(), true);
+
+ // Verify that the large file has been deleted
+ files = listFiles(table);
+ assertThat(files).doesNotContain(largeFileName);
+ }
+
+ private void deleteFile(TableLoader tableLoader, String fileName, boolean expectSuccess)
Review Comment:
We usually try to avoid having boolean parameters, because it is hard to read the code without visiting the method declaration and explicitly checking the parameter.
We sometimes accept this in test code, but please leave a comment where they are called. Something like this:
```
deleteFile(tableLoader(), nonexistentFile, true /* expectSuccess */);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#discussion_r2287550358
##########
flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/DeleteFilesProcessor.java:
##########
@@ -76,6 +79,10 @@ public void open() throws Exception {
taskMetricGroup.counter(TableMaintenanceMetrics.DELETE_FILE_FAILED_COUNTER);
this.succeededCounter =
taskMetricGroup.counter(TableMaintenanceMetrics.DELETE_FILE_SUCCEEDED_COUNTER);
+ this.deleteFileTimeMsHistogram =
+ taskMetricGroup.histogram(
+ TableMaintenanceMetrics.DELETE_FILE_TIME_MS_HISTOGRAM,
+ new DescriptiveStatisticsHistogram(1000));
Review Comment:
@pvary @Guosmilesmile We are currently investigating the integration of the ODS layer with Iceberg, though we do not yet have a concrete business scenario. Nevertheless, I believe adding a metric for the duration of delete operations is valuable: it has very low overhead but can significantly enhance system observability, helping us identify potential bottlenecks and analyze performance trends. For example, if delete times increase abnormally, we can further investigate whether the issue stems from the underlying object storage/HDFS performance or slower HMS responses. In contrast, relying solely on success/failure counts is insufficient to fully reflect the system’s state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#discussion_r2295303950
##########
flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestDeleteFilesProcessor.java:
##########
@@ -74,27 +79,172 @@ void testDeleteMissingFile() throws Exception {
Path dummyFile =
FileSystems.getDefault().getPath(table.location().substring(5), DUMMY_FILE_NAME);
- deleteFile(tableLoader(), dummyFile.toString());
+ deleteFile(tableLoader(), dummyFile.toString(), true);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
@Test
void testInvalidURIScheme() throws Exception {
- deleteFile(tableLoader(), "wrong://");
+ deleteFile(tableLoader(), "wrong://", false);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
- private void deleteFile(TableLoader tableLoader, String fileName) throws Exception {
- tableLoader().open();
+ @Test
+ void testDeleteNonExistentFile() throws Exception {
+ String nonexistentFile = "nonexistentFile.txt";
+
+ deleteFile(tableLoader(), nonexistentFile, true);
+
+ assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
+ }
+
+ @Test
+ void testDeleteLargeFile() throws Exception {
Review Comment:
@pvary This is a very good question! In most practical scenarios, the metadata files of Iceberg tables are usually not very large. However, considering our internal use cases, I wrote this unit test to validate potential risks.
We plan to provide feature data for the company’s internal AI team. According to user feedback, these feature tables may contain hundreds of columns and require support for multiple concurrent writing tasks. This combination of high dimensionality and high concurrency could cause metadata files to grow rapidly.
Therefore, we need to strike a balance between two strategies:
- More, smaller metadata files — beneficial for concurrent writes but increases metadata management overhead.
- Fewer, larger metadata files — reduces the number of files but may increase latency for individual operations.
Currently, the default value of MANIFEST_TARGET_SIZE_BYTES in our system is 8 MB:
```
public static final String MANIFEST_TARGET_SIZE_BYTES = "commit.manifest.target-size-bytes";
public static final long MANIFEST_TARGET_SIZE_BYTES_DEFAULT = 8 * 1024 * 1024; // 8 MB
```
We are now evaluating the appropriate value for this parameter. One suggestion is to increase it to 256 MB. Next, we will conduct performance tests to compare throughput and latency under different configurations, so that we can determine the optimal setting.
I’d really like to hear about your experiences with metadata management. If you could share some practical approaches and insights, that would be fantastic.
cc: @mxm @Guosmilesmile
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#discussion_r2295311375
##########
flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/DeleteFilesProcessor.java:
##########
@@ -116,6 +124,23 @@ private void deleteFiles() {
e);
succeededCounter.inc(deletedFilesCount);
failedCounter.inc(e.numberFailedObjects());
+ } finally {
+ long elapsed = System.currentTimeMillis() - startTime;
+ if (deleteFileTimeMsHistogram != null) {
Review Comment:
You're right, this check isn’t necessary here. I’ll improve this part of the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#discussion_r2295439694
##########
flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestDeleteFilesProcessor.java:
##########
@@ -74,27 +79,172 @@ void testDeleteMissingFile() throws Exception {
Path dummyFile =
FileSystems.getDefault().getPath(table.location().substring(5), DUMMY_FILE_NAME);
- deleteFile(tableLoader(), dummyFile.toString());
+ deleteFile(tableLoader(), dummyFile.toString(), true);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
@Test
void testInvalidURIScheme() throws Exception {
- deleteFile(tableLoader(), "wrong://");
+ deleteFile(tableLoader(), "wrong://", false);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
- private void deleteFile(TableLoader tableLoader, String fileName) throws Exception {
- tableLoader().open();
+ @Test
+ void testDeleteNonExistentFile() throws Exception {
+ String nonexistentFile = "nonexistentFile.txt";
+
+ deleteFile(tableLoader(), nonexistentFile, true);
+
+ assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
+ }
+
+ @Test
+ void testDeleteLargeFile() throws Exception {
+ // Simulate a large file (e.g., 100MB file)
+ String largeFileName = "largeFile.txt";
+ Path largeFile = Path.of(tablePath(table).toString(), largeFileName);
+
+ // Write a large file to disk (this will simulate the large file in the filesystem)
+ byte[] largeData = new byte[1024 * 1024 * 100]; // 100 MB
+ Files.write(largeFile, largeData);
+
+ // Verify that the file was created
+ Set<String> files = listFiles(table);
+ assertThat(files).contains(largeFileName);
+
+ // Use the DeleteFilesProcessor to delete the large file
+ deleteFile(tableLoader(), largeFile.toString(), true);
+
+ // Verify that the large file has been deleted
+ files = listFiles(table);
+ assertThat(files).doesNotContain(largeFileName);
+ }
+
+ private void deleteFile(TableLoader tableLoader, String fileName, boolean expectSuccess)
+ throws Exception {
+ tableLoader.open();
+ DeleteFilesProcessor deleteFilesProcessor =
+ new DeleteFilesProcessor(table, DUMMY_TASK_NAME, 0, 10);
try (OneInputStreamOperatorTestHarness<String, Void> testHarness =
- new OneInputStreamOperatorTestHarness<>(
- new DeleteFilesProcessor(table, DUMMY_TASK_NAME, 0, 10), StringSerializer.INSTANCE)) {
+ new OneInputStreamOperatorTestHarness<>(deleteFilesProcessor, StringSerializer.INSTANCE)) {
testHarness.open();
testHarness.processElement(fileName, System.currentTimeMillis());
testHarness.processWatermark(EVENT_TIME);
testHarness.endInput();
+
+ // Validate if the metrics meet expectations
+ if (expectSuccess) {
+ assertThat(deleteFilesProcessor.getSucceededCounter().getCount()).isEqualTo(1);
+ assertThat(deleteFilesProcessor.getFailedCounter().getCount()).isEqualTo(0);
+ assertThat(deleteFilesProcessor.getDeleteFileTimeMsHistogram().getStatistics().getMean())
+ .isGreaterThan(0);
+ } else {
+ assertThat(deleteFilesProcessor.getSucceededCounter().getCount()).isEqualTo(0);
+ assertThat(deleteFilesProcessor.getFailedCounter().getCount()).isEqualTo(1);
+ assertThat(deleteFilesProcessor.getDeleteFileTimeMsHistogram().getStatistics().getMean())
+ .isGreaterThan(0);
+ }
+
+ } finally {
+ deleteFilesProcessor.close();
+ }
+ }
+
+ @Test
+ void testBatchDelete() throws Exception {
+ // Simulate adding multiple files
+ Set<String> filesToDelete = Sets.newHashSet(TABLE_FILES);
+ filesToDelete.add("file1.txt");
+ filesToDelete.add("file2.txt");
+
+ // Use a smaller batch size to trigger batch deletion logic
+ DeleteFilesProcessor deleteFilesProcessor =
+ new DeleteFilesProcessor(table, DUMMY_TASK_NAME, 0, 2);
+ try (OneInputStreamOperatorTestHarness<String, Void> testHarness =
+ new OneInputStreamOperatorTestHarness<>(deleteFilesProcessor, StringSerializer.INSTANCE)) {
+ testHarness.open();
+
+ for (String file : filesToDelete) {
+ testHarness.processElement(file, System.currentTimeMillis());
+ }
+
+ testHarness.processWatermark(EVENT_TIME);
+ testHarness.endInput();
+
+ // Verify that files are deleted
+ assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
+ assertThat(deleteFilesProcessor.getSucceededCounter().getCount())
+ .isEqualTo(filesToDelete.size());
+ assertThat(deleteFilesProcessor.getFailedCounter().getCount()).isEqualTo(0);
+ assertThat(deleteFilesProcessor.getDeleteFileTimeMsHistogram().getStatistics().getMean())
+ .isGreaterThan(0);
+ } finally {
+ deleteFilesProcessor.close();
+ }
+ }
+
+ @Test
+ void testConcurrentDelete() throws Exception {
Review Comment:
> How often this delete actually happens concurrently?
> Like one file deleted by the `DeleteFilesProcessor`, and one file deleted by the `executorService`?
You are right — the way I previously simulated concurrency may not reflect a realistic scenario. Instead, what we really want to validate is the case where multiple tasks operate on the same table at the same time, leading to concurrent deletions. To cover this, I have improved the unit test by using two DeleteFilesProcessor instances deleting files concurrently.
> If we want to test real concurrency, we likely need to use more files, and maybe some delays.
> If we are ok with checking concurrently deleted files, then `testDeleteNonExistentFile` might be enough
In the latest test, I ensured that both processors attempt to delete the same files concurrently, which allows us to verify that DeleteFilesProcessor behaves as expected under concurrent conditions.
From my local run, I can observe both processors deleting files at the same time:
```
[deleter-p2] INFO org.apache.iceberg.flink.maintenance.operator.DeleteFilesProcessor - Deleted 2 files from table hadoop.default.t using bulk deletes
[deleter-p1] INFO org.apache.iceberg.flink.maintenance.operator.DeleteFilesProcessor - Deleted 2 files from table hadoop.default.t using bulk deletes
[deleter-p1] INFO org.apache.iceberg.flink.maintenance.operator.DeleteFilesProcessor - Deleted 2 files from table hadoop.default.t using bulk deletes
[deleter-p2] INFO org.apache.iceberg.flink.maintenance.operator.DeleteFilesProcessor - Deleted 2 files from table hadoop.default.t using bulk deletes
[deleter-p2] INFO org.apache.iceberg.flink.maintenance.operator.DeleteFilesProcessor - Deleted 2 files from table hadoop.default.t using bulk deletes
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#discussion_r2287550358
##########
flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/DeleteFilesProcessor.java:
##########
@@ -76,6 +79,10 @@ public void open() throws Exception {
taskMetricGroup.counter(TableMaintenanceMetrics.DELETE_FILE_FAILED_COUNTER);
this.succeededCounter =
taskMetricGroup.counter(TableMaintenanceMetrics.DELETE_FILE_SUCCEEDED_COUNTER);
+ this.deleteFileTimeMsHistogram =
+ taskMetricGroup.histogram(
+ TableMaintenanceMetrics.DELETE_FILE_TIME_MS_HISTOGRAM,
+ new DescriptiveStatisticsHistogram(1000));
Review Comment:
@pvary @Guosmilesmile We are currently investigating the integration of the ODS layer with Iceberg, though we do not yet have a concrete business scenario. Nevertheless, I believe adding a metric for the duration of delete operations is valuable: it introduces very low overhead but can significantly enhance system observability, helping us identify potential bottlenecks and analyze performance trends. For example, if delete times increase abnormally, we can further investigate whether the issue stems from the underlying object storage/HDFS performance or slower HMS responses.
In this sense, latency histograms provide a more sensitive signal than simply tracking success/failure counts. Even if deletes succeed, a spike in latency can quickly reveal hidden storage or service bottlenecks, which would otherwise be difficult to detect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#discussion_r2312488793
##########
flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/DeleteFilesProcessor.java:
##########
@@ -116,6 +124,21 @@ private void deleteFiles() {
e);
succeededCounter.inc(deletedFilesCount);
failedCounter.inc(e.numberFailedObjects());
+ } finally {
+ long elapsed = System.currentTimeMillis() - startTime;
+ deleteFileTimeMsHistogram.update(elapsed);
}
}
+
+ public Counter getFailedCounter() {
+ return failedCounter;
+ }
+
+ public Counter getSucceededCounter() {
+ return succeededCounter;
+ }
+
+ public Histogram getDeleteFileTimeMsHistogram() {
+ return deleteFileTimeMsHistogram;
+ }
Review Comment:
> Please remove the get from the method name. See: https://iceberg.apache.org/contribute/#method-naming
> It might also worth to read through the other style guides as well.
> Also these methods should not be public. They should be package private and should be annotated with
> @VisibleForTesting annotation. Check the other cases in the Flink code
Thank you for the information! I have improved the code based on your suggestions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#discussion_r2298142793
##########
flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestDeleteFilesProcessor.java:
##########
@@ -74,27 +79,183 @@ void testDeleteMissingFile() throws Exception {
Path dummyFile =
FileSystems.getDefault().getPath(table.location().substring(5), DUMMY_FILE_NAME);
- deleteFile(tableLoader(), dummyFile.toString());
+ deleteFile(tableLoader(), dummyFile.toString(), true /* expectSuccess */);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
@Test
void testInvalidURIScheme() throws Exception {
- deleteFile(tableLoader(), "wrong://");
+ deleteFile(tableLoader(), "wrong://", false /* expectFail */);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
- private void deleteFile(TableLoader tableLoader, String fileName) throws Exception {
- tableLoader().open();
+ @Test
+ void testDeleteNonExistentFile() throws Exception {
+ String nonexistentFile = "nonexistentFile.txt";
+
+ deleteFile(tableLoader(), nonexistentFile, true /* expectSuccess */);
+
+ assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
+ }
+
+ @Test
+ void testDelete10MBFile() throws Exception {
+ // Simulate a large file (e.g., 10MB file)
+ String largeFileName = "largeFile.txt";
+ Path largeFile = Path.of(tablePath(table).toString(), largeFileName);
+
+ // Write a large file to disk (this will simulate the large file in the filesystem)
+ byte[] largeData = new byte[1024 * 1024 * 10]; // 10 MB
+ Files.write(largeFile, largeData);
+
+ // Verify that the file was created
+ Set<String> files = listFiles(table);
+ assertThat(files).contains(largeFileName);
+
+ // Use the DeleteFilesProcessor to delete the large file
+ deleteFile(tableLoader(), largeFile.toString(), true /* expectSuccess */);
+
+ // Verify that the large file has been deleted
+ files = listFiles(table);
+ assertThat(files).doesNotContain(largeFileName);
+ }
+
+ @Test
+ void testBatchDelete() throws Exception {
+ // Simulate adding multiple files
+ Set<String> filesToDelete = Sets.newHashSet(TABLE_FILES);
+ filesToDelete.add("file1.txt");
+ filesToDelete.add("file2.txt");
+
+ // Use a smaller batch size to trigger batch deletion logic
+ DeleteFilesProcessor deleteFilesProcessor =
+ new DeleteFilesProcessor(table, DUMMY_TASK_NAME, 0, 2);
try (OneInputStreamOperatorTestHarness<String, Void> testHarness =
- new OneInputStreamOperatorTestHarness<>(
- new DeleteFilesProcessor(table, DUMMY_TASK_NAME, 0, 10), StringSerializer.INSTANCE)) {
+ new OneInputStreamOperatorTestHarness<>(deleteFilesProcessor, StringSerializer.INSTANCE)) {
testHarness.open();
- testHarness.processElement(fileName, System.currentTimeMillis());
+
+ for (String file : filesToDelete) {
+ testHarness.processElement(file, System.currentTimeMillis());
+ }
+
+ testHarness.processWatermark(EVENT_TIME);
+ testHarness.endInput();
+
+ // Verify that files are deleted
+ assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
+ assertThat(deleteFilesProcessor.getSucceededCounter().getCount())
+ .isEqualTo(filesToDelete.size());
+ assertThat(deleteFilesProcessor.getFailedCounter().getCount()).isEqualTo(0);
+ assertThat(deleteFilesProcessor.getDeleteFileTimeMsHistogram().getStatistics().getMean())
+ .isGreaterThan(0);
+ } finally {
+ deleteFilesProcessor.close();
+ }
+ }
+
+ @Test
+ void testConcurrentDelete() throws Exception {
+ Path root = tablePath(table);
+
+ // Generate 30 test files: delete-0.txt ... delete-29.txt
+ Set<String> targets = Sets.newHashSet();
+ for (int i = 0; i < 30; i++) {
+ targets.add("delete-" + i + ".txt");
+ }
+
+ for (String f : targets) {
+ Files.write(root.resolve(f), f.getBytes(StandardCharsets.UTF_8));
+ }
+ assertThat(listFiles(table)).containsAll(targets);
+
+ DeleteFilesProcessor p1 = new DeleteFilesProcessor(table, DUMMY_TASK_NAME + "-p1", 0, 2);
+ DeleteFilesProcessor p2 = new DeleteFilesProcessor(table, DUMMY_TASK_NAME + "-p2", 0, 2);
+
+ // Two processors that will try to delete the same files concurrently
+ try (OneInputStreamOperatorTestHarness<String, Void> h1 =
+ new OneInputStreamOperatorTestHarness<>(p1, StringSerializer.INSTANCE);
+ OneInputStreamOperatorTestHarness<String, Void> h2 =
+ new OneInputStreamOperatorTestHarness<>(p2, StringSerializer.INSTANCE)) {
+ h1.open();
+ h2.open();
+
+ // One barrier per file: ensures p1 and p2 try to delete the same file at the same time
+ Map<String, CyclicBarrier> barriers = Maps.newHashMap();
+ targets.forEach(f -> barriers.put(f, new CyclicBarrier(2)));
+
+ long ts = System.currentTimeMillis();
+
+ Thread t1 =
+ new Thread(
+ () -> {
+ try {
+ for (String f : targets) {
+ barriers.get(f).await(2, TimeUnit.SECONDS);
+ h1.processElement(f, ts);
+ }
+ h1.processWatermark(EVENT_TIME);
+ h1.endInput();
+ } catch (Exception ignored) {
+ }
+ },
+ "deleter-p1");
Review Comment:
Do we have to do these in threads? Could we just advance the processors in a loop?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#issuecomment-3392637848
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#discussion_r2300604588
##########
flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/DeleteFilesProcessor.java:
##########
@@ -116,6 +124,21 @@ private void deleteFiles() {
e);
succeededCounter.inc(deletedFilesCount);
failedCounter.inc(e.numberFailedObjects());
+ } finally {
+ long elapsed = System.currentTimeMillis() - startTime;
+ deleteFileTimeMsHistogram.update(elapsed);
}
}
+
+ public Counter getFailedCounter() {
+ return failedCounter;
+ }
+
+ public Counter getSucceededCounter() {
+ return succeededCounter;
+ }
+
+ public Histogram getDeleteFileTimeMsHistogram() {
+ return deleteFileTimeMsHistogram;
+ }
Review Comment:
Thank you for reviewing the code. I will make improvements based on your suggestions as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#discussion_r2282586826
##########
flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestDeleteFilesProcessor.java:
##########
@@ -74,27 +79,172 @@ void testDeleteMissingFile() throws Exception {
Path dummyFile =
FileSystems.getDefault().getPath(table.location().substring(5), DUMMY_FILE_NAME);
- deleteFile(tableLoader(), dummyFile.toString());
+ deleteFile(tableLoader(), dummyFile.toString(), true);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
@Test
void testInvalidURIScheme() throws Exception {
- deleteFile(tableLoader(), "wrong://");
+ deleteFile(tableLoader(), "wrong://", false);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
- private void deleteFile(TableLoader tableLoader, String fileName) throws Exception {
- tableLoader().open();
+ @Test
+ void testDeleteNonExistentFile() throws Exception {
+ String nonexistentFile = "nonexistentFile.txt";
+
+ deleteFile(tableLoader(), nonexistentFile, true);
+
+ assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
+ }
+
+ @Test
+ void testDeleteLargeFile() throws Exception {
+ // Simulate a large file (e.g., 100MB file)
+ String largeFileName = "largeFile.txt";
+ Path largeFile = Path.of(tablePath(table).toString(), largeFileName);
+
+ // Write a large file to disk (this will simulate the large file in the filesystem)
+ byte[] largeData = new byte[1024 * 1024 * 100]; // 100 MB
+ Files.write(largeFile, largeData);
+
+ // Verify that the file was created
+ Set<String> files = listFiles(table);
+ assertThat(files).contains(largeFileName);
+
+ // Use the DeleteFilesProcessor to delete the large file
+ deleteFile(tableLoader(), largeFile.toString(), true);
+
+ // Verify that the large file has been deleted
+ files = listFiles(table);
+ assertThat(files).doesNotContain(largeFileName);
+ }
+
+ private void deleteFile(TableLoader tableLoader, String fileName, boolean expectSuccess)
+ throws Exception {
Review Comment:
Can we put this private method at the end of the file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#discussion_r2282576393
##########
flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/DeleteFilesProcessor.java:
##########
@@ -116,6 +124,23 @@ private void deleteFiles() {
e);
succeededCounter.inc(deletedFilesCount);
failedCounter.inc(e.numberFailedObjects());
+ } finally {
+ long elapsed = System.currentTimeMillis() - startTime;
+ if (deleteFileTimeMsHistogram != null) {
Review Comment:
Is this check needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#discussion_r2298135927
##########
flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestDeleteFilesProcessor.java:
##########
@@ -74,27 +79,183 @@ void testDeleteMissingFile() throws Exception {
Path dummyFile =
FileSystems.getDefault().getPath(table.location().substring(5), DUMMY_FILE_NAME);
- deleteFile(tableLoader(), dummyFile.toString());
+ deleteFile(tableLoader(), dummyFile.toString(), true /* expectSuccess */);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
@Test
void testInvalidURIScheme() throws Exception {
- deleteFile(tableLoader(), "wrong://");
+ deleteFile(tableLoader(), "wrong://", false /* expectFail */);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
- private void deleteFile(TableLoader tableLoader, String fileName) throws Exception {
- tableLoader().open();
+ @Test
+ void testDeleteNonExistentFile() throws Exception {
+ String nonexistentFile = "nonexistentFile.txt";
+
+ deleteFile(tableLoader(), nonexistentFile, true /* expectSuccess */);
+
+ assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
+ }
+
+ @Test
+ void testDelete10MBFile() throws Exception {
+ // Simulate a large file (e.g., 10MB file)
+ String largeFileName = "largeFile.txt";
+ Path largeFile = Path.of(tablePath(table).toString(), largeFileName);
+
+ // Write a large file to disk (this will simulate the large file in the filesystem)
+ byte[] largeData = new byte[1024 * 1024 * 10]; // 10 MB
+ Files.write(largeFile, largeData);
+
+ // Verify that the file was created
+ Set<String> files = listFiles(table);
+ assertThat(files).contains(largeFileName);
+
+ // Use the DeleteFilesProcessor to delete the large file
+ deleteFile(tableLoader(), largeFile.toString(), true /* expectSuccess */);
+
+ // Verify that the large file has been deleted
+ files = listFiles(table);
+ assertThat(files).doesNotContain(largeFileName);
+ }
+
+ @Test
+ void testBatchDelete() throws Exception {
+ // Simulate adding multiple files
+ Set<String> filesToDelete = Sets.newHashSet(TABLE_FILES);
+ filesToDelete.add("file1.txt");
+ filesToDelete.add("file2.txt");
+
+ // Use a smaller batch size to trigger batch deletion logic
+ DeleteFilesProcessor deleteFilesProcessor =
+ new DeleteFilesProcessor(table, DUMMY_TASK_NAME, 0, 2);
try (OneInputStreamOperatorTestHarness<String, Void> testHarness =
- new OneInputStreamOperatorTestHarness<>(
- new DeleteFilesProcessor(table, DUMMY_TASK_NAME, 0, 10), StringSerializer.INSTANCE)) {
+ new OneInputStreamOperatorTestHarness<>(deleteFilesProcessor, StringSerializer.INSTANCE)) {
testHarness.open();
- testHarness.processElement(fileName, System.currentTimeMillis());
+
+ for (String file : filesToDelete) {
+ testHarness.processElement(file, System.currentTimeMillis());
+ }
+
+ testHarness.processWatermark(EVENT_TIME);
+ testHarness.endInput();
+
+ // Verify that files are deleted
+ assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
+ assertThat(deleteFilesProcessor.getSucceededCounter().getCount())
+ .isEqualTo(filesToDelete.size());
+ assertThat(deleteFilesProcessor.getFailedCounter().getCount()).isEqualTo(0);
+ assertThat(deleteFilesProcessor.getDeleteFileTimeMsHistogram().getStatistics().getMean())
+ .isGreaterThan(0);
+ } finally {
+ deleteFilesProcessor.close();
+ }
+ }
+
+ @Test
+ void testConcurrentDelete() throws Exception {
+ Path root = tablePath(table);
+
+ // Generate 30 test files: delete-0.txt ... delete-29.txt
+ Set<String> targets = Sets.newHashSet();
+ for (int i = 0; i < 30; i++) {
+ targets.add("delete-" + i + ".txt");
+ }
+
+ for (String f : targets) {
+ Files.write(root.resolve(f), f.getBytes(StandardCharsets.UTF_8));
+ }
Review Comment:
Why not merge the 2 `for` loops?
Also add newline after every block. See: https://iceberg.apache.org/contribute/#block-spacing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#discussion_r2318687305
##########
flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/DeleteFilesProcessor.java:
##########
@@ -76,6 +79,10 @@ public void open() throws Exception {
taskMetricGroup.counter(TableMaintenanceMetrics.DELETE_FILE_FAILED_COUNTER);
this.succeededCounter =
taskMetricGroup.counter(TableMaintenanceMetrics.DELETE_FILE_SUCCEEDED_COUNTER);
+ this.deleteFileTimeMsHistogram =
+ taskMetricGroup.histogram(
+ TableMaintenanceMetrics.DELETE_FILE_TIME_MS_HISTOGRAM,
+ new DescriptiveStatisticsHistogram(1000));
Review Comment:
@mxm Thank you very much for your suggestion! I have made the changes to the code based on your suggestions. Could you please take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#discussion_r2319383501
##########
flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestDeleteFilesProcessor.java:
##########
@@ -74,27 +75,158 @@ void testDeleteMissingFile() throws Exception {
Path dummyFile =
FileSystems.getDefault().getPath(table.location().substring(5), DUMMY_FILE_NAME);
- deleteFile(tableLoader(), dummyFile.toString());
+ deleteFile(tableLoader(), dummyFile.toString(), true /* expectSuccess */);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
@Test
void testInvalidURIScheme() throws Exception {
- deleteFile(tableLoader(), "wrong://");
+ deleteFile(tableLoader(), "wrong://", false /* expectFail */);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
- private void deleteFile(TableLoader tableLoader, String fileName) throws Exception {
- tableLoader().open();
+ @Test
+ void testDeleteNonExistentFile() throws Exception {
+ String nonexistentFile = "nonexistentFile.txt";
+
+ deleteFile(tableLoader(), nonexistentFile, true /* expectSuccess */);
+
+ assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
+ }
+
+ @Test
+ void testDelete10MBFile() throws Exception {
+ // Simulate a large file (e.g., 10MB file)
+ String largeFileName = "largeFile.txt";
+ Path largeFile = Path.of(tablePath(table).toString(), largeFileName);
+
+ // Write a large file to disk (this will simulate the large file in the filesystem)
+ byte[] largeData = new byte[1024 * 1024 * 10]; // 10 MB
+ Files.write(largeFile, largeData);
+
+ // Verify that the file was created
+ Set<String> files = listFiles(table);
+ assertThat(files).contains(largeFileName);
+
+ // Use the DeleteFilesProcessor to delete the large file
+ deleteFile(tableLoader(), largeFile.toString(), true /* expectSuccess */);
+
+ // Verify that the large file has been deleted
+ files = listFiles(table);
+ assertThat(files).doesNotContain(largeFileName);
+ }
Review Comment:
Apologies for the inconvenience. Since I am responsible for the development of part of the storage software at the company, I’ve been using a fixed testing approach (such as deleting large files), but these methods may not be fully compatible with the current scenario. I agree with your point that this unit test is unnecessary, and I have already removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#discussion_r2287244841
##########
flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/DeleteFilesProcessor.java:
##########
@@ -76,6 +79,10 @@ public void open() throws Exception {
taskMetricGroup.counter(TableMaintenanceMetrics.DELETE_FILE_FAILED_COUNTER);
this.succeededCounter =
taskMetricGroup.counter(TableMaintenanceMetrics.DELETE_FILE_SUCCEEDED_COUNTER);
+ this.deleteFileTimeMsHistogram =
+ taskMetricGroup.histogram(
+ TableMaintenanceMetrics.DELETE_FILE_TIME_MS_HISTOGRAM,
+ new DescriptiveStatisticsHistogram(1000));
Review Comment:
In my use case, I haven't focused on the time it takes to delete files. The only time I've encountered long deletion times is when IO is high, and in those cases, write performance or machine metrics would directly reflect the issue. @slfan1989 Could you describe the specific scenario you're dealing with?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#discussion_r2282579397
##########
flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestDeleteFilesProcessor.java:
##########
@@ -74,27 +79,172 @@ void testDeleteMissingFile() throws Exception {
Path dummyFile =
FileSystems.getDefault().getPath(table.location().substring(5), DUMMY_FILE_NAME);
- deleteFile(tableLoader(), dummyFile.toString());
+ deleteFile(tableLoader(), dummyFile.toString(), true);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
@Test
void testInvalidURIScheme() throws Exception {
- deleteFile(tableLoader(), "wrong://");
+ deleteFile(tableLoader(), "wrong://", false);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
- private void deleteFile(TableLoader tableLoader, String fileName) throws Exception {
- tableLoader().open();
+ @Test
+ void testDeleteNonExistentFile() throws Exception {
+ String nonexistentFile = "nonexistentFile.txt";
+
+ deleteFile(tableLoader(), nonexistentFile, true);
+
+ assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
+ }
+
+ @Test
+ void testDeleteLargeFile() throws Exception {
Review Comment:
I don't understand this test case?
Why would deleting a large file effect anything?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#discussion_r2295429458
##########
flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestDeleteFilesProcessor.java:
##########
@@ -74,27 +79,172 @@ void testDeleteMissingFile() throws Exception {
Path dummyFile =
FileSystems.getDefault().getPath(table.location().substring(5), DUMMY_FILE_NAME);
- deleteFile(tableLoader(), dummyFile.toString());
+ deleteFile(tableLoader(), dummyFile.toString(), true);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
@Test
void testInvalidURIScheme() throws Exception {
- deleteFile(tableLoader(), "wrong://");
+ deleteFile(tableLoader(), "wrong://", false);
assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
}
- private void deleteFile(TableLoader tableLoader, String fileName) throws Exception {
- tableLoader().open();
+ @Test
+ void testDeleteNonExistentFile() throws Exception {
+ String nonexistentFile = "nonexistentFile.txt";
+
+ deleteFile(tableLoader(), nonexistentFile, true);
+
+ assertThat(listFiles(table)).isEqualTo(TABLE_FILES);
+ }
+
+ @Test
+ void testDeleteLargeFile() throws Exception {
+ // Simulate a large file (e.g., 100MB file)
+ String largeFileName = "largeFile.txt";
+ Path largeFile = Path.of(tablePath(table).toString(), largeFileName);
+
+ // Write a large file to disk (this will simulate the large file in the filesystem)
+ byte[] largeData = new byte[1024 * 1024 * 100]; // 100 MB
+ Files.write(largeFile, largeData);
+
+ // Verify that the file was created
+ Set<String> files = listFiles(table);
+ assertThat(files).contains(largeFileName);
+
+ // Use the DeleteFilesProcessor to delete the large file
+ deleteFile(tableLoader(), largeFile.toString(), true);
+
+ // Verify that the large file has been deleted
+ files = listFiles(table);
+ assertThat(files).doesNotContain(largeFileName);
+ }
+
+ private void deleteFile(TableLoader tableLoader, String fileName, boolean expectSuccess)
Review Comment:
Thanks for the suggestion! I’ll add comments to this function and mark the parameters at the call sites the way you recommended.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#discussion_r2298131764
##########
flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/DeleteFilesProcessor.java:
##########
@@ -116,6 +124,21 @@ private void deleteFiles() {
e);
succeededCounter.inc(deletedFilesCount);
failedCounter.inc(e.numberFailedObjects());
+ } finally {
+ long elapsed = System.currentTimeMillis() - startTime;
+ deleteFileTimeMsHistogram.update(elapsed);
}
}
+
+ public Counter getFailedCounter() {
+ return failedCounter;
+ }
+
+ public Counter getSucceededCounter() {
+ return succeededCounter;
+ }
+
+ public Histogram getDeleteFileTimeMsHistogram() {
+ return deleteFileTimeMsHistogram;
+ }
Review Comment:
Please remove the `get` from the method name. See: https://iceberg.apache.org/contribute/#method-naming
It might also worth to read through the other style guides as well.
Also these methods should not be public. They should be package private and should be annotated with `@VisibleForTesting` annotation. Check the other cases in the Flink code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Enhance DeleteFilesProcessor with Time Metrics and Unit Tests. [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13831:
URL: https://github.com/apache/iceberg/pull/13831#issuecomment-3367594445
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org