[GitHub] [iceberg] openinx commented on a change in pull request #1704: Flink : Fix Repeated Rewrite for RewriteDataFilesAction

[GitBox <gi...@apache.org>]

openinx commented on a change in pull request #1704:
URL: https://github.com/apache/iceberg/pull/1704#discussion_r549636084



##########
File path: flink/src/test/java/org/apache/iceberg/flink/actions/TestRewriteDataFilesAction.java
##########
@@ -280,4 +292,82 @@ public void testRewriteLargeTableHasResiduals() throws IOException {
     // Assert the table records as expected.
     SimpleDataUtil.assertTableRecords(icebergTableUnPartitioned, expected);
   }
+
+  /**
+   * a test case to test avoid repeate compress
+   * <p>
+   * If datafile cannot be combined to CombinedScanTask with other DataFiles, the size of the CombinedScanTask list size
+   * is 1, so we remove these CombinedScanTasks to avoid compressed repeatedly.
+   * <p>
+   * In this test case,we generated 3 data files and set targetSizeInBytes greater than the largest file size so that it
+   * cannot be  combined a CombinedScanTask with other datafiles. The datafile with the largest file size will not be
+   * compressed.
+   *
+   * @throws IOException IOException
+   */
+  @Test
+  public void testRewriteAvoidRepeateCompress() throws IOException {
+    Assume.assumeFalse("ORC does not support getting length when file is opening", format.equals(FileFormat.ORC));
+    List<Record> expected = Lists.newArrayList();
+    Schema schema = icebergTableUnPartitioned.schema();
+    GenericAppenderFactory genericAppenderFactory = new GenericAppenderFactory(schema);
+    File file = temp.newFile();
+    int count = 0;
+    try (FileAppender<Record> fileAppender = genericAppenderFactory.newAppender(Files.localOutput(file), format)) {
+      long filesize = 20000;
+      for (; fileAppender.length() < filesize; count++) {
+        Record record = RECORD.copy();
+        record.setField("id", count);
+        record.setField("data", "iceberg");

Review comment:
       nit: we could just use the the: 
   
   ```java
   Record record = SimpleDataUtil.createRecord(count, "iceberg");
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org