You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/10/17 07:23:10 UTC

[GitHub] [iceberg] zhangjun0x01 opened a new pull request #1623: Add a rewrite datafile action for flink

zhangjun0x01 opened a new pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623


   this pr is for #1610 
   
   Add a rewrite datafile action for flink,extracted some common functions to iceberg-core so that we can refactor RewriteDataFilesAction of spark later to avoid duplication of code


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] openinx commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r507547639



##########
File path: flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.RowDataTaskWriterFactory;
+import org.apache.iceberg.flink.sink.TaskWriterFactory;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.LocationProvider;
+import org.apache.iceberg.io.TaskWriter;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
+
+public class RowDataRewriter implements Serializable {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RowDataRewriter.class);
+
+  private final Schema schema;
+  private final PartitionSpec spec;
+  private final Map<String, String> properties;
+  private final FileFormat format;
+  private final LocationProvider locations;
+  private final String nameMapping;
+  private final FileIO io;
+  private final boolean caseSensitive;
+  private final EncryptionManager encryptionManager;
+  private final Table table;
+
+  private final TaskWriterFactory<RowData> taskWriterFactory;
+
+  public RowDataRewriter(Table table, PartitionSpec spec, boolean caseSensitive,
+                         FileIO io, EncryptionManager encryptionManager) {
+    this.table = table;
+    this.schema = table.schema();
+    this.spec = spec;
+    this.locations = table.locationProvider();
+    this.properties = table.properties();
+    this.io = io;
+    this.encryptionManager = encryptionManager;
+
+    this.caseSensitive = caseSensitive;
+    this.nameMapping = table.properties().get(DEFAULT_NAME_MAPPING);
+
+    String formatString = table.properties().getOrDefault(
+        TableProperties.DEFAULT_FILE_FORMAT, TableProperties.DEFAULT_FILE_FORMAT_DEFAULT);
+    this.format = FileFormat.valueOf(formatString.toUpperCase(Locale.ENGLISH));
+
+    RowType flinkSchema = FlinkSchemaUtil.convert(table.schema());
+    taskWriterFactory = new RowDataTaskWriterFactory(table.schema(),
+        flinkSchema,
+        table.spec(),
+        table.locationProvider(),
+        table.io(),
+        table.encryption(),
+        Long.MAX_VALUE,

Review comment:
       The `targetFileSizeBytes` will always be unlimited ?  Pls set a configuration keys for it,  [here](https://github.com/apache/iceberg/pull/1083/files#r438890925) is the discussion we had for spark compaction.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] openinx commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r507567915



##########
File path: flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.RowDataTaskWriterFactory;
+import org.apache.iceberg.flink.sink.TaskWriterFactory;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.LocationProvider;
+import org.apache.iceberg.io.TaskWriter;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
+
+public class RowDataRewriter implements Serializable {

Review comment:
       Does this class need to be serializable ?  I checked the code, seems it won't need to be encoded or decoded for network transferring ? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] openinx commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r514774682



##########
File path: core/src/main/java/org/apache/iceberg/actions/BaseRewriteDataFilesAction.java
##########
@@ -268,7 +272,7 @@ private void replaceDataFiles(Iterable<DataFile> deletedDataFiles, Iterable<Data
     }
   }
 
-  protected abstract FileIO fileIO();
+  protected abstract FileIO setFileIO();

Review comment:
       why `setFileIO` ?  That's quite confusing,  it's more likely we are getting fileIO while we name it as `setFileIO` ?  I don't think we need to change this. 

##########
File path: flink/src/main/java/org/apache/iceberg/actions/Actions.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.actions;

Review comment:
       I'm thinking that we may need to move this class into `org.apache.iceberg.flink.actions` as we flink & spark both have the same class name `Actions` under the package `org.apache.iceberg.actions`,  then how to identify the class if someone want to execute the actions in the same project for both flink and spark ? 

##########
File path: flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.stream.Collectors;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.RowDataTaskWriterFactory;
+import org.apache.iceberg.flink.sink.TaskWriterFactory;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.TaskWriter;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
+
+public class RowDataRewriter {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RowDataRewriter.class);
+
+  private final Schema schema;
+  private final FileFormat format;
+  private final String nameMapping;
+  private final FileIO io;
+  private final boolean caseSensitive;
+  private final EncryptionManager encryptionManager;
+  private final TaskWriterFactory<RowData> taskWriterFactory;
+
+  public RowDataRewriter(Table table, boolean caseSensitive, FileIO io, EncryptionManager encryptionManager) {
+    this.schema = table.schema();
+    this.io = io;
+    this.encryptionManager = encryptionManager;
+    this.caseSensitive = caseSensitive;
+    this.nameMapping = PropertyUtil.propertyAsString(table.properties(), DEFAULT_NAME_MAPPING, null);
+    String formatString = PropertyUtil.propertyAsString(table.properties(), TableProperties.DEFAULT_FILE_FORMAT,

Review comment:
       nit: could we separate this `format` parser into a separate code block by left a new empty line ? That makes the code more clear.

##########
File path: flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.stream.Collectors;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.RowDataTaskWriterFactory;
+import org.apache.iceberg.flink.sink.TaskWriterFactory;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.TaskWriter;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
+
+public class RowDataRewriter {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RowDataRewriter.class);
+
+  private final Schema schema;
+  private final FileFormat format;
+  private final String nameMapping;
+  private final FileIO io;
+  private final boolean caseSensitive;
+  private final EncryptionManager encryptionManager;
+  private final TaskWriterFactory<RowData> taskWriterFactory;
+
+  public RowDataRewriter(Table table, boolean caseSensitive, FileIO io, EncryptionManager encryptionManager) {
+    this.schema = table.schema();
+    this.io = io;
+    this.encryptionManager = encryptionManager;
+    this.caseSensitive = caseSensitive;
+    this.nameMapping = PropertyUtil.propertyAsString(table.properties(), DEFAULT_NAME_MAPPING, null);
+    String formatString = PropertyUtil.propertyAsString(table.properties(), TableProperties.DEFAULT_FILE_FORMAT,
+        TableProperties.DEFAULT_FILE_FORMAT_DEFAULT);
+    this.format = FileFormat.valueOf(formatString.toUpperCase(Locale.ENGLISH));
+    RowType flinkSchema = FlinkSchemaUtil.convert(table.schema());
+    this.taskWriterFactory = new RowDataTaskWriterFactory(
+        table.schema(),
+        flinkSchema,
+        table.spec(),
+        table.locationProvider(),
+        io,
+        encryptionManager,
+        Long.MAX_VALUE,
+        format,
+        table.properties());
+  }
+
+  public List<DataFile> rewriteDataForTasks(DataSet<CombinedScanTask> dataSet) throws Exception {
+    RewriteMap map = new RewriteMap(schema, nameMapping, io, caseSensitive, encryptionManager, taskWriterFactory);
+    DataSet<List<DataFile>> ds = dataSet.map(map);
+    return ds.collect().stream().flatMap(Collection::stream).collect(Collectors.toList());
+  }
+
+  public static class RewriteMap extends RichMapFunction<CombinedScanTask, List<DataFile>> {
+
+    private transient TaskWriter<RowData> writer;
+    private transient int subTaskId;
+    private transient int attemptId;
+
+    private final Schema schema;
+    private final String nameMapping;
+    private final FileIO io;
+    private final boolean caseSensitive;
+    private final EncryptionManager encryptionManager;
+    private final TaskWriterFactory<RowData> taskWriterFactory;
+
+    public RewriteMap(Schema schema, String nameMapping, FileIO io, boolean caseSensitive,
+                      EncryptionManager encryptionManager, TaskWriterFactory<RowData> taskWriterFactory) {
+      this.schema = schema;
+      this.nameMapping = nameMapping;
+      this.io = io;
+      this.caseSensitive = caseSensitive;
+      this.encryptionManager = encryptionManager;
+      this.taskWriterFactory = taskWriterFactory;
+    }
+
+    @Override
+    public void open(Configuration parameters) {
+      this.subTaskId = getRuntimeContext().getIndexOfThisSubtask();
+      this.attemptId = getRuntimeContext().getAttemptNumber();
+      // Initialize the task writer factory.
+      this.taskWriterFactory.initialize(subTaskId, attemptId);
+    }
+
+    @Override
+    public List<DataFile> map(CombinedScanTask task) throws Exception {
+      // Initialize the task writer.
+      this.writer = taskWriterFactory.create();
+      RowDataIterator iterator =
+          new RowDataIterator(task, io, encryptionManager, schema, schema, nameMapping, caseSensitive);
+      try {
+        while (iterator.hasNext()) {
+          RowData rowData = iterator.next();
+          writer.write(rowData);
+        }
+        iterator.close();
+        return Lists.newArrayList(writer.complete());
+      } catch (Throwable originalThrowable) {
+        try {

Review comment:
       I think we've forgot to close the `RowDataIterator` if encountered any exception when iterate the records.  Pls handle this carefully because if close failed then we won't want to close it again in the `catch` block.

##########
File path: flink/src/test/java/org/apache/iceberg/actions/TestRewriteDataFilesAction.java
##########
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+package org.apache.iceberg.actions;
+
+import java.util.List;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.flink.FlinkCatalogTestBase;
+import org.apache.iceberg.flink.SimpleDataUtil;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+@RunWith(Parameterized.class)
+public class TestRewriteDataFilesAction extends FlinkCatalogTestBase {

Review comment:
       I think we should also provide an unit test similar to the spark `testRewriteLargeTableHasResiduals`, that test the option `ignoreResiduals`.  

##########
File path: flink/src/main/java/org/apache/iceberg/actions/RewriteDataFilesAction.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.actions;
+
+import java.util.List;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.source.RowDataRewriter;
+import org.apache.iceberg.io.FileIO;
+
+public class RewriteDataFilesAction extends

Review comment:
       nit: don't have to wrap the line here ..

##########
File path: flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.stream.Collectors;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.RowDataTaskWriterFactory;
+import org.apache.iceberg.flink.sink.TaskWriterFactory;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.TaskWriter;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
+
+public class RowDataRewriter {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RowDataRewriter.class);
+
+  private final Schema schema;
+  private final FileFormat format;
+  private final String nameMapping;
+  private final FileIO io;
+  private final boolean caseSensitive;
+  private final EncryptionManager encryptionManager;
+  private final TaskWriterFactory<RowData> taskWriterFactory;
+
+  public RowDataRewriter(Table table, boolean caseSensitive, FileIO io, EncryptionManager encryptionManager) {
+    this.schema = table.schema();
+    this.io = io;
+    this.encryptionManager = encryptionManager;
+    this.caseSensitive = caseSensitive;
+    this.nameMapping = PropertyUtil.propertyAsString(table.properties(), DEFAULT_NAME_MAPPING, null);
+    String formatString = PropertyUtil.propertyAsString(table.properties(), TableProperties.DEFAULT_FILE_FORMAT,
+        TableProperties.DEFAULT_FILE_FORMAT_DEFAULT);
+    this.format = FileFormat.valueOf(formatString.toUpperCase(Locale.ENGLISH));

Review comment:
       nit: could we keep the same order between the constructor's argument assignment and fields definition ?  for example:
   
   ```java
   private final int a;
   private final int b;
   
   public Construct(int a, int b){
       this.a = a;
       this.b = b;
   }
   
   ```
   

##########
File path: core/src/main/java/org/apache/iceberg/util/PropertyUtil.java
##########
@@ -52,4 +52,13 @@ public static long propertyAsLong(Map<String, String> properties,
     }
     return defaultValue;
   }
+
+  public static String propertyAsString(Map<String, String> properties,
+                                    String property, String defaultValue) {

Review comment:
       nit:  code format ? 

##########
File path: flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.stream.Collectors;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.RowDataTaskWriterFactory;
+import org.apache.iceberg.flink.sink.TaskWriterFactory;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.TaskWriter;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
+
+public class RowDataRewriter {

Review comment:
       we don't need to expose this class to public ? As the apache iceberg is a lib,  so we're tried to avoid to expose unnecessary internal classes or interfaces to public, so that users won't abuse them. 

##########
File path: flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.stream.Collectors;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.RowDataTaskWriterFactory;
+import org.apache.iceberg.flink.sink.TaskWriterFactory;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.TaskWriter;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
+
+public class RowDataRewriter {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RowDataRewriter.class);
+
+  private final Schema schema;
+  private final FileFormat format;
+  private final String nameMapping;
+  private final FileIO io;
+  private final boolean caseSensitive;
+  private final EncryptionManager encryptionManager;
+  private final TaskWriterFactory<RowData> taskWriterFactory;
+
+  public RowDataRewriter(Table table, boolean caseSensitive, FileIO io, EncryptionManager encryptionManager) {
+    this.schema = table.schema();
+    this.io = io;
+    this.encryptionManager = encryptionManager;
+    this.caseSensitive = caseSensitive;
+    this.nameMapping = PropertyUtil.propertyAsString(table.properties(), DEFAULT_NAME_MAPPING, null);
+    String formatString = PropertyUtil.propertyAsString(table.properties(), TableProperties.DEFAULT_FILE_FORMAT,
+        TableProperties.DEFAULT_FILE_FORMAT_DEFAULT);
+    this.format = FileFormat.valueOf(formatString.toUpperCase(Locale.ENGLISH));
+    RowType flinkSchema = FlinkSchemaUtil.convert(table.schema());
+    this.taskWriterFactory = new RowDataTaskWriterFactory(
+        table.schema(),
+        flinkSchema,
+        table.spec(),
+        table.locationProvider(),
+        io,
+        encryptionManager,
+        Long.MAX_VALUE,
+        format,
+        table.properties());
+  }
+
+  public List<DataFile> rewriteDataForTasks(DataSet<CombinedScanTask> dataSet) throws Exception {
+    RewriteMap map = new RewriteMap(schema, nameMapping, io, caseSensitive, encryptionManager, taskWriterFactory);
+    DataSet<List<DataFile>> ds = dataSet.map(map);
+    return ds.collect().stream().flatMap(Collection::stream).collect(Collectors.toList());
+  }
+
+  public static class RewriteMap extends RichMapFunction<CombinedScanTask, List<DataFile>> {
+
+    private transient TaskWriter<RowData> writer;

Review comment:
       nit:  could just use local variable ?  don't have to be a transient field.

##########
File path: flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.stream.Collectors;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.RowDataTaskWriterFactory;
+import org.apache.iceberg.flink.sink.TaskWriterFactory;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.TaskWriter;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
+
+public class RowDataRewriter {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RowDataRewriter.class);
+
+  private final Schema schema;
+  private final FileFormat format;
+  private final String nameMapping;
+  private final FileIO io;
+  private final boolean caseSensitive;
+  private final EncryptionManager encryptionManager;
+  private final TaskWriterFactory<RowData> taskWriterFactory;
+
+  public RowDataRewriter(Table table, boolean caseSensitive, FileIO io, EncryptionManager encryptionManager) {
+    this.schema = table.schema();
+    this.io = io;
+    this.encryptionManager = encryptionManager;
+    this.caseSensitive = caseSensitive;
+    this.nameMapping = PropertyUtil.propertyAsString(table.properties(), DEFAULT_NAME_MAPPING, null);
+    String formatString = PropertyUtil.propertyAsString(table.properties(), TableProperties.DEFAULT_FILE_FORMAT,
+        TableProperties.DEFAULT_FILE_FORMAT_DEFAULT);
+    this.format = FileFormat.valueOf(formatString.toUpperCase(Locale.ENGLISH));
+    RowType flinkSchema = FlinkSchemaUtil.convert(table.schema());
+    this.taskWriterFactory = new RowDataTaskWriterFactory(
+        table.schema(),
+        flinkSchema,
+        table.spec(),
+        table.locationProvider(),
+        io,
+        encryptionManager,
+        Long.MAX_VALUE,
+        format,
+        table.properties());
+  }
+
+  public List<DataFile> rewriteDataForTasks(DataSet<CombinedScanTask> dataSet) throws Exception {
+    RewriteMap map = new RewriteMap(schema, nameMapping, io, caseSensitive, encryptionManager, taskWriterFactory);
+    DataSet<List<DataFile>> ds = dataSet.map(map);
+    return ds.collect().stream().flatMap(Collection::stream).collect(Collectors.toList());
+  }
+
+  public static class RewriteMap extends RichMapFunction<CombinedScanTask, List<DataFile>> {
+
+    private transient TaskWriter<RowData> writer;
+    private transient int subTaskId;
+    private transient int attemptId;
+
+    private final Schema schema;
+    private final String nameMapping;
+    private final FileIO io;
+    private final boolean caseSensitive;
+    private final EncryptionManager encryptionManager;
+    private final TaskWriterFactory<RowData> taskWriterFactory;
+
+    public RewriteMap(Schema schema, String nameMapping, FileIO io, boolean caseSensitive,
+                      EncryptionManager encryptionManager, TaskWriterFactory<RowData> taskWriterFactory) {
+      this.schema = schema;
+      this.nameMapping = nameMapping;
+      this.io = io;
+      this.caseSensitive = caseSensitive;
+      this.encryptionManager = encryptionManager;
+      this.taskWriterFactory = taskWriterFactory;
+    }
+
+    @Override
+    public void open(Configuration parameters) {
+      this.subTaskId = getRuntimeContext().getIndexOfThisSubtask();
+      this.attemptId = getRuntimeContext().getAttemptNumber();
+      // Initialize the task writer factory.
+      this.taskWriterFactory.initialize(subTaskId, attemptId);
+    }
+
+    @Override
+    public List<DataFile> map(CombinedScanTask task) throws Exception {
+      // Initialize the task writer.
+      this.writer = taskWriterFactory.create();
+      RowDataIterator iterator =
+          new RowDataIterator(task, io, encryptionManager, schema, schema, nameMapping, caseSensitive);
+      try {
+        while (iterator.hasNext()) {
+          RowData rowData = iterator.next();
+          writer.write(rowData);
+        }
+        iterator.close();
+        return Lists.newArrayList(writer.complete());
+      } catch (Throwable originalThrowable) {
+        try {
+          LOG.error("Aborting commit for  (subTaskId {}, attemptId {})", subTaskId, attemptId);
+          writer.abort();
+          LOG.error("Aborted commit for  (subTaskId {}, attemptId {})", subTaskId, attemptId);
+        } catch (Throwable inner) {
+          if (originalThrowable != inner) {
+            originalThrowable.addSuppressed(inner);

Review comment:
       What's the reason that we need to supress the `inner` exception when it's different with the `originalThrowable` ? 

##########
File path: flink/src/test/java/org/apache/iceberg/actions/TestRewriteDataFilesAction.java
##########
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+package org.apache.iceberg.actions;
+
+import java.util.List;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.flink.FlinkCatalogTestBase;
+import org.apache.iceberg.flink.SimpleDataUtil;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+@RunWith(Parameterized.class)
+public class TestRewriteDataFilesAction extends FlinkCatalogTestBase {
+
+  private static final String TABLE_NAME_UNPARTIITONED = "test_table_unpartitioned";
+  private static final String TABLE_NAME_PARTIITONED = "test_table_partitioned";
+  private final FileFormat format;
+  private Table icebergTableUnPartitioned;
+  private Table icebergTablePartitioned;
+
+  public TestRewriteDataFilesAction(String catalogName, String[] baseNamespace, FileFormat format) {
+    super(catalogName, baseNamespace);
+    this.format = format;
+  }
+
+  @Parameterized.Parameters(name = "catalogName={0}, baseNamespace={1}, format={2}")
+  public static Iterable<Object[]> parameters() {
+    List<Object[]> parameters = Lists.newArrayList();
+    for (FileFormat format : new FileFormat[] {FileFormat.ORC, FileFormat.AVRO, FileFormat.PARQUET}) {
+      for (Object[] catalogParams : FlinkCatalogTestBase.parameters()) {
+        String catalogName = (String) catalogParams[0];
+        String[] baseNamespace = (String[]) catalogParams[1];
+        parameters.add(new Object[] {catalogName, baseNamespace, format});
+      }
+    }
+    return parameters;
+  }
+
+  @Before
+  public void before() {
+    super.before();
+    sql("CREATE DATABASE %s", flinkDatabase);
+    sql("USE CATALOG %s", catalogName);
+    sql("USE %s", DATABASE);
+    sql("CREATE TABLE %s (id int, data varchar) with ('write.format.default'='%s')", TABLE_NAME_UNPARTIITONED,
+        format.name());
+    icebergTableUnPartitioned = validationCatalog.loadTable(TableIdentifier.of(icebergNamespace,
+        TABLE_NAME_UNPARTIITONED));
+
+    sql("CREATE TABLE %s (id int, data varchar)  PARTITIONED BY (data) with ('write.format.default'='%s')",
+        TABLE_NAME_PARTIITONED, format.name());
+    icebergTablePartitioned = validationCatalog.loadTable(TableIdentifier.of(icebergNamespace,
+        TABLE_NAME_PARTIITONED));
+  }
+
+  @After
+  public void clean() {
+    sql("DROP TABLE IF EXISTS %s.%s", flinkDatabase, TABLE_NAME_UNPARTIITONED);
+    sql("DROP TABLE IF EXISTS %s.%s", flinkDatabase, TABLE_NAME_PARTIITONED);
+    sql("DROP DATABASE IF EXISTS %s", flinkDatabase);
+    super.clean();
+  }
+
+  @Test
+  public void testRewriteDataFilesEmptyTable() throws Exception {
+    Assert.assertNull("Table must be empty", icebergTableUnPartitioned.currentSnapshot());
+    Actions.forTable(icebergTableUnPartitioned)
+        .rewriteDataFiles()
+        .execute();
+    Assert.assertNull("Table must stay empty", icebergTableUnPartitioned.currentSnapshot());
+  }
+
+
+  @Test
+  public void testRewriteDataFilesUnpartitionedTable() throws Exception {
+    sql("INSERT INTO %s SELECT 1, 'hello'", TABLE_NAME_UNPARTIITONED);
+    sql("INSERT INTO %s SELECT 2, 'world'", TABLE_NAME_UNPARTIITONED);
+
+    icebergTableUnPartitioned.refresh();
+
+    CloseableIterable<FileScanTask> tasks = icebergTableUnPartitioned.newScan().planFiles();
+    List<DataFile> dataFiles = Lists.newArrayList(CloseableIterable.transform(tasks, FileScanTask::file));
+    Assert.assertEquals("Should have 2 data files before rewrite", 2, dataFiles.size());
+
+    RewriteDataFilesActionResult result =
+        Actions.forTable(icebergTableUnPartitioned)
+            .rewriteDataFiles()
+            .execute();
+
+    Assert.assertEquals("Action should rewrite 2 data files", 2, result.deletedDataFiles().size());
+    Assert.assertEquals("Action should add 1 data file", 1, result.addedDataFiles().size());
+
+    icebergTableUnPartitioned.refresh();
+
+    CloseableIterable<FileScanTask> tasks1 = icebergTableUnPartitioned.newScan().planFiles();
+    List<DataFile> dataFiles1 = Lists.newArrayList(CloseableIterable.transform(tasks1, FileScanTask::file));
+    Assert.assertEquals("Should have 1 data files after rewrite", 1, dataFiles1.size());
+
+    // Assert the table records as expected.
+    SimpleDataUtil.assertTableRecords(icebergTableUnPartitioned, Lists.newArrayList(
+        SimpleDataUtil.createRecord(1, "hello"),
+        SimpleDataUtil.createRecord(2, "world")
+    ));
+  }
+
+  @Test
+  public void testRewriteDataFilesPartitionedTable() throws Exception {
+    sql("INSERT INTO %s SELECT 1, 'hello' ", TABLE_NAME_PARTIITONED);
+    sql("INSERT INTO %s SELECT 2, 'hello' ", TABLE_NAME_PARTIITONED);
+    sql("INSERT INTO %s SELECT 3, 'world' ", TABLE_NAME_PARTIITONED);
+    sql("INSERT INTO %s SELECT 4, 'world' ", TABLE_NAME_PARTIITONED);
+
+    icebergTablePartitioned.refresh();
+
+    CloseableIterable<FileScanTask> tasks = icebergTablePartitioned.newScan().planFiles();
+    List<DataFile> dataFiles = Lists.newArrayList(CloseableIterable.transform(tasks, FileScanTask::file));
+    Assert.assertEquals("Should have 4 data files before rewrite", 4, dataFiles.size());
+
+    RewriteDataFilesActionResult result =
+        Actions.forTable(icebergTablePartitioned)
+            .rewriteDataFiles()
+            .execute();
+
+    Assert.assertEquals("Action should rewrite 4 data files", 4, result.deletedDataFiles().size());
+    Assert.assertEquals("Action should add 2 data file", 2, result.addedDataFiles().size());
+
+    icebergTablePartitioned.refresh();
+
+    CloseableIterable<FileScanTask> tasks1 = icebergTablePartitioned.newScan().planFiles();
+    List<DataFile> dataFiles1 = Lists.newArrayList(CloseableIterable.transform(tasks1, FileScanTask::file));
+    Assert.assertEquals("Should have 2 data files after rewrite", 2, dataFiles1.size());
+
+    // Assert the table records as expected.
+    SimpleDataUtil.assertTableRecords(icebergTablePartitioned, Lists.newArrayList(
+        SimpleDataUtil.createRecord(1, "hello"),
+        SimpleDataUtil.createRecord(2, "hello"),
+        SimpleDataUtil.createRecord(3, "world"),
+        SimpleDataUtil.createRecord(4, "world")
+    ));
+  }
+
+
+  @Test
+  public void testRewriteDataFilesWithFilter() throws Exception {
+    sql("INSERT INTO %s SELECT 1, 'hello' ", TABLE_NAME_PARTIITONED);
+    sql("INSERT INTO %s SELECT 1, 'hello' ", TABLE_NAME_PARTIITONED);
+    sql("INSERT INTO %s SELECT 1, 'world' ", TABLE_NAME_PARTIITONED);
+    sql("INSERT INTO %s SELECT 2, 'world' ", TABLE_NAME_PARTIITONED);
+    sql("INSERT INTO %s SELECT 3, 'world' ", TABLE_NAME_PARTIITONED);
+
+    icebergTablePartitioned.refresh();
+
+    CloseableIterable<FileScanTask> tasks = icebergTablePartitioned.newScan().planFiles();
+    List<DataFile> dataFiles = Lists.newArrayList(CloseableIterable.transform(tasks, FileScanTask::file));
+    Assert.assertEquals("Should have 5 data files before rewrite", 5, dataFiles.size());
+
+    RewriteDataFilesActionResult result =
+        Actions.forTable(icebergTablePartitioned)
+            .rewriteDataFiles()
+            .filter(Expressions.equal("id", 1))
+            .filter(Expressions.startsWith("data", "he"))
+            .execute();
+
+    Assert.assertEquals("Action should rewrite 2 data files", 2, result.deletedDataFiles().size());
+    Assert.assertEquals("Action should add 1 data file", 1, result.addedDataFiles().size());
+
+    icebergTablePartitioned.refresh();
+
+    CloseableIterable<FileScanTask> tasks1 = icebergTablePartitioned.newScan().planFiles();
+    List<DataFile> dataFiles1 = Lists.newArrayList(CloseableIterable.transform(tasks1, FileScanTask::file));
+    Assert.assertEquals("Should have 4 data files after rewrite", 4, dataFiles1.size());
+
+    // Assert the table records as expected.
+    SimpleDataUtil.assertTableRecords(icebergTablePartitioned, Lists.newArrayList(

Review comment:
       I think we may need to refactor the `SimpleDataUtil.assertTableRecords`, because it will use `Set` to check the expected records, that means it will de-duplicate the same records automatically, and won't identify the case that have two `<1, 'hello'>` records. 

##########
File path: flink/src/test/java/org/apache/iceberg/flink/FlinkTestBase.java
##########
@@ -72,7 +72,7 @@ protected TableEnvironment getTableEnv() {
     return tEnv;
   }
 
-  List<Object[]> sql(String query, Object... args) {
+  public List<Object[]> sql(String query, Object... args) {

Review comment:
       How about use `protected` ? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] openinx commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r507591327



##########
File path: flink/src/main/java/org/apache/iceberg/actions/RewriteDataFilesAction.java
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.actions;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.source.RowDataRewriter;
+import org.apache.iceberg.util.StructLikeWrapper;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class RewriteDataFilesAction extends
+    RewriteDataFilesActionBase<RewriteDataFilesAction, RewriteDataFilesActionResult> {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RewriteDataFilesAction.class);
+
+  private ExecutionEnvironment env = null;
+  private boolean caseSensitive;
+
+  public RewriteDataFilesAction(ExecutionEnvironment env, Table table) {
+    super(table);
+    this.env = env;
+    caseSensitive = Boolean.parseBoolean(table.properties().getOrDefault("case-sensitive", "false"));
+  }
+
+
+  @Override
+  public RewriteDataFilesActionResult execute() {
+    Map<StructLikeWrapper, Collection<FileScanTask>> filteredGroupedTasks = getFilteredGroupedTasks();
+
+    // Nothing to rewrite if there's only one DataFile in each partition.
+    if (filteredGroupedTasks.isEmpty()) {
+      return RewriteDataFilesActionResult.empty();
+    }
+
+    List<DataFile> currentDataFiles = getCurrentDataFiles(filteredGroupedTasks);
+
+    List<CombinedScanTask> combinedScanTasks = getCombinedScanTasks(filteredGroupedTasks);
+
+    DataSet<CombinedScanTask> dataSet = env.fromCollection(combinedScanTasks);
+    env.setParallelism(combinedScanTasks.size());
+
+    RowDataRewriter rowDataRewriter =
+        new RowDataRewriter(getTable(), getTable().spec(), caseSensitive, getTable().io(), getEncryptionManager());
+
+    List<DataFile> addedDataFiles = null;
+    try {
+      addedDataFiles = rowDataRewriter.rewriteDataForTasks(dataSet);
+    } catch (Exception e) {
+      LOG.error("rewrite data files error", e);

Review comment:
       Both `RuntimeException`  and `UncheckedIOException` works fine for me. Actually,  `UncheckedIOException` is also a `RuntimeException`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] openinx commented on pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
openinx commented on pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#issuecomment-717688387


   @zhangjun0x01 Could you rebase this patch to resolve the conflicts ?  I will take another look for this, Thanks. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] zhangjun0x01 commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
zhangjun0x01 commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r507599871



##########
File path: flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.RowDataTaskWriterFactory;
+import org.apache.iceberg.flink.sink.TaskWriterFactory;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.LocationProvider;
+import org.apache.iceberg.io.TaskWriter;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
+
+public class RowDataRewriter implements Serializable {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RowDataRewriter.class);
+
+  private final Schema schema;
+  private final PartitionSpec spec;
+  private final Map<String, String> properties;
+  private final FileFormat format;
+  private final LocationProvider locations;
+  private final String nameMapping;
+  private final FileIO io;
+  private final boolean caseSensitive;
+  private final EncryptionManager encryptionManager;
+  private final Table table;
+
+  private final TaskWriterFactory<RowData> taskWriterFactory;
+
+  public RowDataRewriter(Table table, PartitionSpec spec, boolean caseSensitive,
+                         FileIO io, EncryptionManager encryptionManager) {
+    this.table = table;
+    this.schema = table.schema();
+    this.spec = spec;
+    this.locations = table.locationProvider();
+    this.properties = table.properties();
+    this.io = io;
+    this.encryptionManager = encryptionManager;
+
+    this.caseSensitive = caseSensitive;
+    this.nameMapping = table.properties().get(DEFAULT_NAME_MAPPING);
+
+    String formatString = table.properties().getOrDefault(
+        TableProperties.DEFAULT_FILE_FORMAT, TableProperties.DEFAULT_FILE_FORMAT_DEFAULT);
+    this.format = FileFormat.valueOf(formatString.toUpperCase(Locale.ENGLISH));
+
+    RowType flinkSchema = FlinkSchemaUtil.convert(table.schema());
+    taskWriterFactory = new RowDataTaskWriterFactory(table.schema(),
+        flinkSchema,
+        table.spec(),
+        table.locationProvider(),
+        table.io(),
+        table.encryption(),
+        Long.MAX_VALUE,
+        format,
+        table.properties());
+  }
+
+
+  public List<DataFile> rewriteDataForTasks(DataSet<CombinedScanTask> dataSet) throws Exception {
+
+    RewriteMap map = new RewriteMap(schema, nameMapping, io, caseSensitive,
+        encryptionManager,
+        taskWriterFactory);
+
+    DataSet<List<DataFile>> ds = dataSet.map(map);
+    return ds.collect().stream()
+        .flatMap(Collection::stream)
+        .collect(Collectors.toList());
+  }
+
+  public static class RewriteMap extends RichMapFunction<CombinedScanTask, List<DataFile>> {
+
+    private transient TaskWriter<RowData> writer;
+    private transient int subTaskId;
+    private transient int attemptId;
+
+    private final Schema schema;
+    private final String nameMapping;
+    private final FileIO io;
+    private final boolean caseSensitive;
+    private final EncryptionManager encryptionManager;
+    private final TaskWriterFactory<RowData> taskWriterFactory;
+
+    public RewriteMap(Schema schema, String nameMapping, FileIO io, boolean caseSensitive,
+                      EncryptionManager encryptionManager,
+                      TaskWriterFactory<RowData> taskWriterFactory) {
+      this.schema = schema;
+      this.nameMapping = nameMapping;
+      this.io = io;
+      this.caseSensitive = caseSensitive;
+      this.encryptionManager = encryptionManager;
+      this.taskWriterFactory = taskWriterFactory;
+    }
+
+    @Override
+    public void open(Configuration parameters) {
+      this.subTaskId = getRuntimeContext().getIndexOfThisSubtask();
+      this.attemptId = getRuntimeContext().getAttemptNumber();
+
+      // Initialize the task writer factory.
+      this.taskWriterFactory.initialize(subTaskId, attemptId);
+
+    }
+
+
+    @Override
+    public List<DataFile> map(CombinedScanTask task) throws Exception {
+      // Initialize the task writer.
+      this.writer = taskWriterFactory.create();
+      RowDataIterator iterator =

Review comment:
       I refer to the spark code, but I didn't find a specific reason for doing so, so I copied it to here . I guess it may have encountered some special exceptions.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] openinx commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r507543365



##########
File path: flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.RowDataTaskWriterFactory;
+import org.apache.iceberg.flink.sink.TaskWriterFactory;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.LocationProvider;
+import org.apache.iceberg.io.TaskWriter;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
+
+public class RowDataRewriter implements Serializable {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RowDataRewriter.class);
+
+  private final Schema schema;
+  private final PartitionSpec spec;
+  private final Map<String, String> properties;
+  private final FileFormat format;
+  private final LocationProvider locations;
+  private final String nameMapping;
+  private final FileIO io;
+  private final boolean caseSensitive;
+  private final EncryptionManager encryptionManager;
+  private final Table table;
+
+  private final TaskWriterFactory<RowData> taskWriterFactory;
+
+  public RowDataRewriter(Table table, PartitionSpec spec, boolean caseSensitive,
+                         FileIO io, EncryptionManager encryptionManager) {
+    this.table = table;
+    this.schema = table.schema();
+    this.spec = spec;
+    this.locations = table.locationProvider();
+    this.properties = table.properties();
+    this.io = io;
+    this.encryptionManager = encryptionManager;
+
+    this.caseSensitive = caseSensitive;
+    this.nameMapping = table.properties().get(DEFAULT_NAME_MAPPING);
+
+    String formatString = table.properties().getOrDefault(

Review comment:
       Maybe it's time to move all the properties parser to a `TablePropsUtil` now.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] openinx commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r507536858



##########
File path: flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.RowDataTaskWriterFactory;
+import org.apache.iceberg.flink.sink.TaskWriterFactory;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.LocationProvider;
+import org.apache.iceberg.io.TaskWriter;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
+
+public class RowDataRewriter implements Serializable {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RowDataRewriter.class);
+
+  private final Schema schema;
+  private final PartitionSpec spec;
+  private final Map<String, String> properties;
+  private final FileFormat format;
+  private final LocationProvider locations;
+  private final String nameMapping;
+  private final FileIO io;
+  private final boolean caseSensitive;
+  private final EncryptionManager encryptionManager;
+  private final Table table;
+
+  private final TaskWriterFactory<RowData> taskWriterFactory;
+
+  public RowDataRewriter(Table table, PartitionSpec spec, boolean caseSensitive,
+                         FileIO io, EncryptionManager encryptionManager) {
+    this.table = table;
+    this.schema = table.schema();
+    this.spec = spec;
+    this.locations = table.locationProvider();
+    this.properties = table.properties();
+    this.io = io;
+    this.encryptionManager = encryptionManager;
+
+    this.caseSensitive = caseSensitive;
+    this.nameMapping = table.properties().get(DEFAULT_NAME_MAPPING);
+
+    String formatString = table.properties().getOrDefault(
+        TableProperties.DEFAULT_FILE_FORMAT, TableProperties.DEFAULT_FILE_FORMAT_DEFAULT);
+    this.format = FileFormat.valueOf(formatString.toUpperCase(Locale.ENGLISH));
+
+    RowType flinkSchema = FlinkSchemaUtil.convert(table.schema());
+    taskWriterFactory = new RowDataTaskWriterFactory(table.schema(),
+        flinkSchema,
+        table.spec(),
+        table.locationProvider(),
+        table.io(),
+        table.encryption(),
+        Long.MAX_VALUE,
+        format,
+        table.properties());
+  }
+
+
+  public List<DataFile> rewriteDataForTasks(DataSet<CombinedScanTask> dataSet) throws Exception {
+
+    RewriteMap map = new RewriteMap(schema, nameMapping, io, caseSensitive,
+        encryptionManager,
+        taskWriterFactory);
+
+    DataSet<List<DataFile>> ds = dataSet.map(map);
+    return ds.collect().stream()
+        .flatMap(Collection::stream)
+        .collect(Collectors.toList());
+  }
+
+  public static class RewriteMap extends RichMapFunction<CombinedScanTask, List<DataFile>> {
+
+    private transient TaskWriter<RowData> writer;
+    private transient int subTaskId;
+    private transient int attemptId;
+
+    private final Schema schema;
+    private final String nameMapping;
+    private final FileIO io;
+    private final boolean caseSensitive;
+    private final EncryptionManager encryptionManager;
+    private final TaskWriterFactory<RowData> taskWriterFactory;
+
+    public RewriteMap(Schema schema, String nameMapping, FileIO io, boolean caseSensitive,
+                      EncryptionManager encryptionManager,
+                      TaskWriterFactory<RowData> taskWriterFactory) {
+      this.schema = schema;
+      this.nameMapping = nameMapping;
+      this.io = io;
+      this.caseSensitive = caseSensitive;
+      this.encryptionManager = encryptionManager;
+      this.taskWriterFactory = taskWriterFactory;
+    }
+
+    @Override
+    public void open(Configuration parameters) {
+      this.subTaskId = getRuntimeContext().getIndexOfThisSubtask();
+      this.attemptId = getRuntimeContext().getAttemptNumber();
+
+      // Initialize the task writer factory.
+      this.taskWriterFactory.initialize(subTaskId, attemptId);
+
+    }
+
+
+    @Override
+    public List<DataFile> map(CombinedScanTask task) throws Exception {
+      // Initialize the task writer.
+      this.writer = taskWriterFactory.create();
+      RowDataIterator iterator =
+          new RowDataIterator(task, io, encryptionManager, schema, schema,
+              nameMapping, caseSensitive);
+      try {
+        while (iterator.hasNext()) {
+          RowData rowData = iterator.next();
+          writer.write(rowData);
+        }
+        iterator.close();
+        writer.close();
+        return Lists.newArrayList(writer.complete()).stream()

Review comment:
       Could we just return  the `Lists.newArrayList(writer.complete())` ? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] zhangjun0x01 commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
zhangjun0x01 commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r507063628



##########
File path: core/src/main/java/org/apache/iceberg/actions/RewriteDataFilesActionBase.java
##########
@@ -0,0 +1,271 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.actions;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RewriteFiles;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.ListMultimap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.relocated.com.google.common.collect.Multimaps;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.relocated.com.google.common.collect.Streams;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.iceberg.util.StructLikeWrapper;
+import org.apache.iceberg.util.TableScanUtil;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public abstract class RewriteDataFilesActionBase<ThisT, R> extends BaseSnapshotUpdateAction<ThisT, R> {

Review comment:
       hi,simonsssu.thanks for your suggestion. I have opened a new PR #1624  and extracted some common logic. I will do rebase in PR #1624 is merged




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] simonsssu commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
simonsssu commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r507053297



##########
File path: core/src/main/java/org/apache/iceberg/actions/RewriteDataFilesActionBase.java
##########
@@ -0,0 +1,271 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.actions;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RewriteFiles;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.ListMultimap;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.relocated.com.google.common.collect.Multimaps;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.relocated.com.google.common.collect.Streams;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.iceberg.util.StructLikeWrapper;
+import org.apache.iceberg.util.TableScanUtil;
+import org.apache.iceberg.util.Tasks;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public abstract class RewriteDataFilesActionBase<ThisT, R> extends BaseSnapshotUpdateAction<ThisT, R> {

Review comment:
       Hi ZhangJun, Thanks for your work. Seems the class only used in Flink and the logic here already exists in Spark part. I think it's better to split this pr into 2 parts, one is refactor current rewrite code and the other is 'add a rewrite action for flink'.

##########
File path: flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.RowDataTaskWriterFactory;
+import org.apache.iceberg.flink.sink.TaskWriterFactory;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.LocationProvider;
+import org.apache.iceberg.io.TaskWriter;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
+
+public class RowDataRewriter implements Serializable {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RowDataRewriter.class);
+
+  private final Schema schema;
+  private final PartitionSpec spec;
+  private final Map<String, String> properties;
+  private final FileFormat format;
+  private final LocationProvider locations;
+  private final String nameMapping;
+  private final FileIO io;
+  private final boolean caseSensitive;
+  private final EncryptionManager encryptionManager;
+  private final Table table;
+
+  private final TaskWriterFactory<RowData> taskWriterFactory;
+
+  public RowDataRewriter(Table table, PartitionSpec spec, boolean caseSensitive,
+                         FileIO io, EncryptionManager encryptionManager) {
+    this.table = table;
+    this.schema = table.schema();
+    this.spec = spec;
+    this.locations = table.locationProvider();
+    this.properties = table.properties();
+    this.io = io;
+    this.encryptionManager = encryptionManager;
+
+    this.caseSensitive = caseSensitive;
+    this.nameMapping = table.properties().get(DEFAULT_NAME_MAPPING);
+
+    String formatString = table.properties().getOrDefault(
+        TableProperties.DEFAULT_FILE_FORMAT, TableProperties.DEFAULT_FILE_FORMAT_DEFAULT);
+    this.format = FileFormat.valueOf(formatString.toUpperCase(Locale.ENGLISH));
+
+    RowType flinkSchema = FlinkSchemaUtil.convert(table.schema());
+    taskWriterFactory = new RowDataTaskWriterFactory(table.schema(),
+        flinkSchema,
+        table.spec(),
+        table.locationProvider(),
+        table.io(),
+        table.encryption(),
+        Long.MAX_VALUE,
+        format,
+        table.properties());
+  }
+
+
+  public List<DataFile> rewriteDataForTasks(DataSet<CombinedScanTask> dataSet) throws Exception {
+
+    RewriteMap map = new RewriteMap(schema, nameMapping, io, caseSensitive,
+        encryptionManager,
+        taskWriterFactory);
+
+    DataSet<List<DataFile>> ds = dataSet.map(map);
+    return ds.collect().stream()
+        .flatMap(Collection::stream)
+        .collect(Collectors.toList());
+  }
+
+  public static class RewriteMap extends RichMapFunction<CombinedScanTask, List<DataFile>> {
+
+    private transient TaskWriter<RowData> writer;
+    private transient int subTaskId;
+    private transient int attemptId;
+
+    private final Schema schema;
+    private final String nameMapping;
+    private final FileIO io;
+    private final boolean caseSensitive;
+    private final EncryptionManager encryptionManager;
+    private final TaskWriterFactory<RowData> taskWriterFactory;
+
+    public RewriteMap(Schema schema, String nameMapping, FileIO io, boolean caseSensitive,
+                      EncryptionManager encryptionManager,
+                      TaskWriterFactory<RowData> taskWriterFactory) {
+      this.schema = schema;
+      this.nameMapping = nameMapping;
+      this.io = io;
+      this.caseSensitive = caseSensitive;
+      this.encryptionManager = encryptionManager;
+      this.taskWriterFactory = taskWriterFactory;
+    }
+
+    @Override
+    public void open(Configuration parameters) {
+      this.subTaskId = getRuntimeContext().getIndexOfThisSubtask();
+      this.attemptId = getRuntimeContext().getAttemptNumber();
+
+      // Initialize the task writer factory.
+      this.taskWriterFactory.initialize(subTaskId, attemptId);
+
+    }
+

Review comment:
       Useless blank line.

##########
File path: flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.RowDataTaskWriterFactory;
+import org.apache.iceberg.flink.sink.TaskWriterFactory;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.LocationProvider;
+import org.apache.iceberg.io.TaskWriter;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
+
+public class RowDataRewriter implements Serializable {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RowDataRewriter.class);
+
+  private final Schema schema;
+  private final PartitionSpec spec;
+  private final Map<String, String> properties;
+  private final FileFormat format;
+  private final LocationProvider locations;
+  private final String nameMapping;
+  private final FileIO io;
+  private final boolean caseSensitive;
+  private final EncryptionManager encryptionManager;
+  private final Table table;
+
+  private final TaskWriterFactory<RowData> taskWriterFactory;
+
+  public RowDataRewriter(Table table, PartitionSpec spec, boolean caseSensitive,
+                         FileIO io, EncryptionManager encryptionManager) {
+    this.table = table;
+    this.schema = table.schema();
+    this.spec = spec;
+    this.locations = table.locationProvider();
+    this.properties = table.properties();
+    this.io = io;
+    this.encryptionManager = encryptionManager;
+
+    this.caseSensitive = caseSensitive;
+    this.nameMapping = table.properties().get(DEFAULT_NAME_MAPPING);
+
+    String formatString = table.properties().getOrDefault(
+        TableProperties.DEFAULT_FILE_FORMAT, TableProperties.DEFAULT_FILE_FORMAT_DEFAULT);
+    this.format = FileFormat.valueOf(formatString.toUpperCase(Locale.ENGLISH));
+
+    RowType flinkSchema = FlinkSchemaUtil.convert(table.schema());
+    taskWriterFactory = new RowDataTaskWriterFactory(table.schema(),
+        flinkSchema,
+        table.spec(),
+        table.locationProvider(),
+        table.io(),
+        table.encryption(),
+        Long.MAX_VALUE,
+        format,
+        table.properties());
+  }
+
+
+  public List<DataFile> rewriteDataForTasks(DataSet<CombinedScanTask> dataSet) throws Exception {
+
+    RewriteMap map = new RewriteMap(schema, nameMapping, io, caseSensitive,
+        encryptionManager,
+        taskWriterFactory);
+
+    DataSet<List<DataFile>> ds = dataSet.map(map);
+    return ds.collect().stream()
+        .flatMap(Collection::stream)
+        .collect(Collectors.toList());
+  }
+
+  public static class RewriteMap extends RichMapFunction<CombinedScanTask, List<DataFile>> {
+
+    private transient TaskWriter<RowData> writer;
+    private transient int subTaskId;
+    private transient int attemptId;
+
+    private final Schema schema;
+    private final String nameMapping;
+    private final FileIO io;
+    private final boolean caseSensitive;
+    private final EncryptionManager encryptionManager;
+    private final TaskWriterFactory<RowData> taskWriterFactory;
+
+    public RewriteMap(Schema schema, String nameMapping, FileIO io, boolean caseSensitive,
+                      EncryptionManager encryptionManager,
+                      TaskWriterFactory<RowData> taskWriterFactory) {
+      this.schema = schema;
+      this.nameMapping = nameMapping;
+      this.io = io;
+      this.caseSensitive = caseSensitive;
+      this.encryptionManager = encryptionManager;
+      this.taskWriterFactory = taskWriterFactory;
+    }
+
+    @Override
+    public void open(Configuration parameters) {
+      this.subTaskId = getRuntimeContext().getIndexOfThisSubtask();
+      this.attemptId = getRuntimeContext().getAttemptNumber();
+
+      // Initialize the task writer factory.
+      this.taskWriterFactory.initialize(subTaskId, attemptId);
+
+    }
+
+
+    @Override
+    public List<DataFile> map(CombinedScanTask task) throws Exception {
+      // Initialize the task writer.
+      this.writer = taskWriterFactory.create();
+      RowDataIterator iterator =

Review comment:
       Can we use a try-with-resource here?  I think it will simply the code to handle exception.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] JingsongLi commented on pull request #1623: Flink: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
JingsongLi commented on pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#issuecomment-721600027


   Hi @zhangjun0x01 , I think we can add a method `setMaxParallelism` to Flink `RewriteDataFilesAction`. It don't need be a table property.
   And I think the default value of maxParallelism can be `env.getParallelism`


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] openinx commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r515916068



##########
File path: flink/src/test/java/org/apache/iceberg/flink/actions/TestRewriteDataFilesAction.java
##########
@@ -0,0 +1,283 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+package org.apache.iceberg.flink.actions;
+
+import java.io.IOException;
+import java.util.List;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.actions.RewriteDataFilesActionResult;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.flink.FlinkCatalogTestBase;
+import org.apache.iceberg.flink.SimpleDataUtil;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.iceberg.flink.SimpleDataUtil.RECORD;
+
+@RunWith(Parameterized.class)
+public class TestRewriteDataFilesAction extends FlinkCatalogTestBase {
+
+  private static final String TABLE_NAME_UNPARTIITONED = "test_table_unpartitioned";

Review comment:
       nit: TABLE_NAME_UNPARTIITONED -> `TABLE_NAME_UNPARTITIONED`. 
   
   `TABLE_NAME_PARTIITONED` has the similar issue.

##########
File path: flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.stream.Collectors;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.RowDataTaskWriterFactory;
+import org.apache.iceberg.flink.sink.TaskWriterFactory;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.TaskWriter;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
+
+public class RowDataRewriter {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RowDataRewriter.class);
+
+  private final Schema schema;
+  private final FileFormat format;
+  private final String nameMapping;
+  private final FileIO io;
+  private final boolean caseSensitive;
+  private final EncryptionManager encryptionManager;
+  private final TaskWriterFactory<RowData> taskWriterFactory;
+
+  public RowDataRewriter(Table table, boolean caseSensitive, FileIO io, EncryptionManager encryptionManager) {
+    this.schema = table.schema();
+    this.caseSensitive = caseSensitive;
+    this.io = io;
+    this.encryptionManager = encryptionManager;
+    this.nameMapping = PropertyUtil.propertyAsString(table.properties(), DEFAULT_NAME_MAPPING, null);
+
+    String formatString = PropertyUtil.propertyAsString(table.properties(), TableProperties.DEFAULT_FILE_FORMAT,
+        TableProperties.DEFAULT_FILE_FORMAT_DEFAULT);
+    this.format = FileFormat.valueOf(formatString.toUpperCase(Locale.ENGLISH));
+    RowType flinkSchema = FlinkSchemaUtil.convert(table.schema());
+    this.taskWriterFactory = new RowDataTaskWriterFactory(
+        table.schema(),
+        flinkSchema,
+        table.spec(),
+        table.locationProvider(),
+        io,
+        encryptionManager,
+        Long.MAX_VALUE,
+        format,
+        table.properties());
+  }
+
+  public List<DataFile> rewriteDataForTasks(DataStream<CombinedScanTask> dataSet, int parallelism) {

Review comment:
       If change to use datastream to submit this job,  then the argument `dataSet` should be renamed to `dataStream` ? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] zhangjun0x01 commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
zhangjun0x01 commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r507547516



##########
File path: flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.RowDataTaskWriterFactory;
+import org.apache.iceberg.flink.sink.TaskWriterFactory;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.LocationProvider;
+import org.apache.iceberg.io.TaskWriter;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
+
+public class RowDataRewriter implements Serializable {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RowDataRewriter.class);
+
+  private final Schema schema;
+  private final PartitionSpec spec;
+  private final Map<String, String> properties;
+  private final FileFormat format;
+  private final LocationProvider locations;
+  private final String nameMapping;
+  private final FileIO io;
+  private final boolean caseSensitive;
+  private final EncryptionManager encryptionManager;
+  private final Table table;
+
+  private final TaskWriterFactory<RowData> taskWriterFactory;
+
+  public RowDataRewriter(Table table, PartitionSpec spec, boolean caseSensitive,
+                         FileIO io, EncryptionManager encryptionManager) {
+    this.table = table;
+    this.schema = table.schema();
+    this.spec = spec;
+    this.locations = table.locationProvider();
+    this.properties = table.properties();
+    this.io = io;
+    this.encryptionManager = encryptionManager;
+
+    this.caseSensitive = caseSensitive;
+    this.nameMapping = table.properties().get(DEFAULT_NAME_MAPPING);
+
+    String formatString = table.properties().getOrDefault(
+        TableProperties.DEFAULT_FILE_FORMAT, TableProperties.DEFAULT_FILE_FORMAT_DEFAULT);
+    this.format = FileFormat.valueOf(formatString.toUpperCase(Locale.ENGLISH));
+
+    RowType flinkSchema = FlinkSchemaUtil.convert(table.schema());
+    taskWriterFactory = new RowDataTaskWriterFactory(table.schema(),
+        flinkSchema,
+        table.spec(),
+        table.locationProvider(),
+        table.io(),
+        table.encryption(),
+        Long.MAX_VALUE,
+        format,
+        table.properties());
+  }
+
+
+  public List<DataFile> rewriteDataForTasks(DataSet<CombinedScanTask> dataSet) throws Exception {
+
+    RewriteMap map = new RewriteMap(schema, nameMapping, io, caseSensitive,
+        encryptionManager,
+        taskWriterFactory);
+
+    DataSet<List<DataFile>> ds = dataSet.map(map);
+    return ds.collect().stream()
+        .flatMap(Collection::stream)
+        .collect(Collectors.toList());
+  }
+
+  public static class RewriteMap extends RichMapFunction<CombinedScanTask, List<DataFile>> {
+
+    private transient TaskWriter<RowData> writer;
+    private transient int subTaskId;
+    private transient int attemptId;
+
+    private final Schema schema;
+    private final String nameMapping;
+    private final FileIO io;
+    private final boolean caseSensitive;
+    private final EncryptionManager encryptionManager;
+    private final TaskWriterFactory<RowData> taskWriterFactory;
+
+    public RewriteMap(Schema schema, String nameMapping, FileIO io, boolean caseSensitive,
+                      EncryptionManager encryptionManager,
+                      TaskWriterFactory<RowData> taskWriterFactory) {
+      this.schema = schema;
+      this.nameMapping = nameMapping;
+      this.io = io;
+      this.caseSensitive = caseSensitive;
+      this.encryptionManager = encryptionManager;
+      this.taskWriterFactory = taskWriterFactory;
+    }
+
+    @Override
+    public void open(Configuration parameters) {
+      this.subTaskId = getRuntimeContext().getIndexOfThisSubtask();
+      this.attemptId = getRuntimeContext().getAttemptNumber();
+
+      // Initialize the task writer factory.
+      this.taskWriterFactory.initialize(subTaskId, attemptId);
+

Review comment:
       I use the iceberg checkstyle to format the code,Is it necessary to add this empty line  limit to checkstyle?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] zhangjun0x01 commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
zhangjun0x01 commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r509857506



##########
File path: flink/src/main/java/org/apache/iceberg/flink/TablePropertiesUtil.java
##########
@@ -0,0 +1,38 @@
+/*

Review comment:
       yes,we should use PropertyUtil, I had delete the TablePropertiesUtil.java ,I didn't find this PropertyUtil at first




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] zhangjun0x01 commented on pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
zhangjun0x01 commented on pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#issuecomment-712076629


   Thank you very much for the review @openinx @simonsssu , I have solved all comments


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] zhangjun0x01 commented on pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
zhangjun0x01 commented on pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#issuecomment-717729357


   @openinx I updated the patch.
   
   As discussed in #1624, the variable fileIO in BaseRewriteDataFilesAction lacks a getter method, so I used BaseRewriteDataFilesAction#fileIO() as its getter method and added a setter method setFileIO().


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] openinx commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r507542189



##########
File path: flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.RowDataTaskWriterFactory;
+import org.apache.iceberg.flink.sink.TaskWriterFactory;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.LocationProvider;
+import org.apache.iceberg.io.TaskWriter;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
+
+public class RowDataRewriter implements Serializable {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RowDataRewriter.class);
+
+  private final Schema schema;
+  private final PartitionSpec spec;
+  private final Map<String, String> properties;
+  private final FileFormat format;
+  private final LocationProvider locations;
+  private final String nameMapping;
+  private final FileIO io;
+  private final boolean caseSensitive;
+  private final EncryptionManager encryptionManager;
+  private final Table table;

Review comment:
       The same thing for `table`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] openinx commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r507539962



##########
File path: flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.RowDataTaskWriterFactory;
+import org.apache.iceberg.flink.sink.TaskWriterFactory;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.LocationProvider;
+import org.apache.iceberg.io.TaskWriter;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
+
+public class RowDataRewriter implements Serializable {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RowDataRewriter.class);
+
+  private final Schema schema;
+  private final PartitionSpec spec;
+  private final Map<String, String> properties;
+  private final FileFormat format;
+  private final LocationProvider locations;
+  private final String nameMapping;
+  private final FileIO io;
+  private final boolean caseSensitive;
+  private final EncryptionManager encryptionManager;
+  private final Table table;
+
+  private final TaskWriterFactory<RowData> taskWriterFactory;
+
+  public RowDataRewriter(Table table, PartitionSpec spec, boolean caseSensitive,
+                         FileIO io, EncryptionManager encryptionManager) {
+    this.table = table;
+    this.schema = table.schema();
+    this.spec = spec;
+    this.locations = table.locationProvider();
+    this.properties = table.properties();
+    this.io = io;
+    this.encryptionManager = encryptionManager;
+
+    this.caseSensitive = caseSensitive;
+    this.nameMapping = table.properties().get(DEFAULT_NAME_MAPPING);
+
+    String formatString = table.properties().getOrDefault(
+        TableProperties.DEFAULT_FILE_FORMAT, TableProperties.DEFAULT_FILE_FORMAT_DEFAULT);
+    this.format = FileFormat.valueOf(formatString.toUpperCase(Locale.ENGLISH));
+
+    RowType flinkSchema = FlinkSchemaUtil.convert(table.schema());
+    taskWriterFactory = new RowDataTaskWriterFactory(table.schema(),
+        flinkSchema,
+        table.spec(),
+        table.locationProvider(),
+        table.io(),
+        table.encryption(),
+        Long.MAX_VALUE,
+        format,
+        table.properties());
+  }
+
+
+  public List<DataFile> rewriteDataForTasks(DataSet<CombinedScanTask> dataSet) throws Exception {
+
+    RewriteMap map = new RewriteMap(schema, nameMapping, io, caseSensitive,
+        encryptionManager,
+        taskWriterFactory);
+
+    DataSet<List<DataFile>> ds = dataSet.map(map);
+    return ds.collect().stream()
+        .flatMap(Collection::stream)
+        .collect(Collectors.toList());
+  }
+
+  public static class RewriteMap extends RichMapFunction<CombinedScanTask, List<DataFile>> {
+
+    private transient TaskWriter<RowData> writer;
+    private transient int subTaskId;
+    private transient int attemptId;
+
+    private final Schema schema;
+    private final String nameMapping;
+    private final FileIO io;
+    private final boolean caseSensitive;
+    private final EncryptionManager encryptionManager;
+    private final TaskWriterFactory<RowData> taskWriterFactory;
+
+    public RewriteMap(Schema schema, String nameMapping, FileIO io, boolean caseSensitive,
+                      EncryptionManager encryptionManager,
+                      TaskWriterFactory<RowData> taskWriterFactory) {
+      this.schema = schema;
+      this.nameMapping = nameMapping;
+      this.io = io;
+      this.caseSensitive = caseSensitive;
+      this.encryptionManager = encryptionManager;
+      this.taskWriterFactory = taskWriterFactory;
+    }
+
+    @Override
+    public void open(Configuration parameters) {
+      this.subTaskId = getRuntimeContext().getIndexOfThisSubtask();
+      this.attemptId = getRuntimeContext().getAttemptNumber();
+
+      // Initialize the task writer factory.
+      this.taskWriterFactory.initialize(subTaskId, attemptId);
+

Review comment:
       nit:  pls don't left a new empty line here. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] openinx commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r507565899



##########
File path: flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.RowDataTaskWriterFactory;
+import org.apache.iceberg.flink.sink.TaskWriterFactory;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.LocationProvider;
+import org.apache.iceberg.io.TaskWriter;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
+
+public class RowDataRewriter implements Serializable {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RowDataRewriter.class);
+
+  private final Schema schema;
+  private final PartitionSpec spec;
+  private final Map<String, String> properties;
+  private final FileFormat format;
+  private final LocationProvider locations;
+  private final String nameMapping;
+  private final FileIO io;
+  private final boolean caseSensitive;
+  private final EncryptionManager encryptionManager;
+  private final Table table;
+
+  private final TaskWriterFactory<RowData> taskWriterFactory;
+
+  public RowDataRewriter(Table table, PartitionSpec spec, boolean caseSensitive,
+                         FileIO io, EncryptionManager encryptionManager) {
+    this.table = table;
+    this.schema = table.schema();
+    this.spec = spec;
+    this.locations = table.locationProvider();
+    this.properties = table.properties();
+    this.io = io;
+    this.encryptionManager = encryptionManager;
+
+    this.caseSensitive = caseSensitive;
+    this.nameMapping = table.properties().get(DEFAULT_NAME_MAPPING);
+
+    String formatString = table.properties().getOrDefault(
+        TableProperties.DEFAULT_FILE_FORMAT, TableProperties.DEFAULT_FILE_FORMAT_DEFAULT);
+    this.format = FileFormat.valueOf(formatString.toUpperCase(Locale.ENGLISH));
+
+    RowType flinkSchema = FlinkSchemaUtil.convert(table.schema());
+    taskWriterFactory = new RowDataTaskWriterFactory(table.schema(),
+        flinkSchema,
+        table.spec(),
+        table.locationProvider(),
+        table.io(),
+        table.encryption(),
+        Long.MAX_VALUE,
+        format,
+        table.properties());
+  }
+
+
+  public List<DataFile> rewriteDataForTasks(DataSet<CombinedScanTask> dataSet) throws Exception {
+
+    RewriteMap map = new RewriteMap(schema, nameMapping, io, caseSensitive,
+        encryptionManager,
+        taskWriterFactory);
+
+    DataSet<List<DataFile>> ds = dataSet.map(map);
+    return ds.collect().stream()
+        .flatMap(Collection::stream)
+        .collect(Collectors.toList());
+  }
+
+  public static class RewriteMap extends RichMapFunction<CombinedScanTask, List<DataFile>> {
+
+    private transient TaskWriter<RowData> writer;
+    private transient int subTaskId;
+    private transient int attemptId;
+
+    private final Schema schema;
+    private final String nameMapping;
+    private final FileIO io;
+    private final boolean caseSensitive;
+    private final EncryptionManager encryptionManager;
+    private final TaskWriterFactory<RowData> taskWriterFactory;
+
+    public RewriteMap(Schema schema, String nameMapping, FileIO io, boolean caseSensitive,
+                      EncryptionManager encryptionManager,
+                      TaskWriterFactory<RowData> taskWriterFactory) {
+      this.schema = schema;
+      this.nameMapping = nameMapping;
+      this.io = io;
+      this.caseSensitive = caseSensitive;
+      this.encryptionManager = encryptionManager;
+      this.taskWriterFactory = taskWriterFactory;
+    }
+
+    @Override
+    public void open(Configuration parameters) {
+      this.subTaskId = getRuntimeContext().getIndexOfThisSubtask();
+      this.attemptId = getRuntimeContext().getAttemptNumber();
+
+      // Initialize the task writer factory.
+      this.taskWriterFactory.initialize(subTaskId, attemptId);
+
+    }
+
+
+    @Override
+    public List<DataFile> map(CombinedScanTask task) throws Exception {
+      // Initialize the task writer.
+      this.writer = taskWriterFactory.create();
+      RowDataIterator iterator =
+          new RowDataIterator(task, io, encryptionManager, schema, schema,
+              nameMapping, caseSensitive);
+      try {
+        while (iterator.hasNext()) {
+          RowData rowData = iterator.next();
+          writer.write(rowData);
+        }
+        iterator.close();
+        writer.close();
+        return Lists.newArrayList(writer.complete()).stream()
+            .collect(Collectors.toList());
+      } catch (Throwable originalThrowable) {
+        try {
+          LOG.error("Aborting task", originalThrowable);
+
+          LOG.error("Aborting commit for  (subTaskId {}, attemptId {})",
+              subTaskId, attemptId);
+          if (iterator != null) {

Review comment:
       nit:  iterator is always not-null ? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] JingsongLi commented on a change in pull request #1623: Flink: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
JingsongLi commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r517215101



##########
File path: core/src/main/java/org/apache/iceberg/actions/BaseRewriteDataFilesAction.java
##########
@@ -268,6 +268,23 @@ private void replaceDataFiles(Iterable<DataFile> deletedDataFiles, Iterable<Data
     }
   }
 
+  protected int getParallelism(int parallelism, List<CombinedScanTask> combinedScanTasks) {

Review comment:
       Spark can handle large parallelism in default...

##########
File path: core/src/main/java/org/apache/iceberg/actions/BaseRewriteDataFilesAction.java
##########
@@ -268,6 +268,23 @@ private void replaceDataFiles(Iterable<DataFile> deletedDataFiles, Iterable<Data
     }
   }
 
+  protected int getParallelism(int parallelism, List<CombinedScanTask> combinedScanTasks) {

Review comment:
       Don't need, Spark can handle large parallelism in default...




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] zhangjun0x01 commented on a change in pull request #1623: Flink: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
zhangjun0x01 commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r517075692



##########
File path: flink/src/main/java/org/apache/iceberg/flink/actions/RewriteDataFilesAction.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.actions;
+
+import java.util.List;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.actions.BaseRewriteDataFilesAction;
+import org.apache.iceberg.flink.source.RowDataRewriter;
+import org.apache.iceberg.io.FileIO;
+
+public class RewriteDataFilesAction extends BaseRewriteDataFilesAction<RewriteDataFilesAction> {
+
+  private ExecutionEnvironment env;
+
+  public RewriteDataFilesAction(ExecutionEnvironment env, Table table) {
+    super(table);
+    this.env = env;
+  }
+
+  @Override
+  protected FileIO fileIO() {
+    return table().io();
+  }
+
+  @Override
+  protected List<DataFile> rewriteDataForTasks(List<CombinedScanTask> combinedScanTasks) {
+    DataSet<CombinedScanTask> dataSet = env.fromCollection(combinedScanTasks).setParallelism(combinedScanTasks.size());

Review comment:
       I  update the code , add the max parallelism in this PR. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] zhangjun0x01 commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
zhangjun0x01 commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r516381086



##########
File path: flink/src/test/java/org/apache/iceberg/flink/actions/TestRewriteDataFilesAction.java
##########
@@ -0,0 +1,283 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+package org.apache.iceberg.flink.actions;
+
+import java.io.IOException;
+import java.util.List;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.actions.RewriteDataFilesActionResult;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.flink.FlinkCatalogTestBase;
+import org.apache.iceberg.flink.SimpleDataUtil;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.iceberg.flink.SimpleDataUtil.RECORD;
+
+@RunWith(Parameterized.class)
+public class TestRewriteDataFilesAction extends FlinkCatalogTestBase {
+
+  private static final String TABLE_NAME_UNPARTIITONED = "test_table_unpartitioned";

Review comment:
       I updated it

##########
File path: flink/src/main/java/org/apache/iceberg/flink/actions/RewriteDataFilesAction.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.actions;
+
+import java.util.List;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.actions.BaseRewriteDataFilesAction;
+import org.apache.iceberg.flink.source.RowDataRewriter;
+import org.apache.iceberg.io.FileIO;
+
+public class RewriteDataFilesAction extends BaseRewriteDataFilesAction<RewriteDataFilesAction> {
+
+  private ExecutionEnvironment env;
+
+  public RewriteDataFilesAction(ExecutionEnvironment env, Table table) {
+    super(table);
+    this.env = env;
+  }
+
+  @Override
+  protected FileIO fileIO() {
+    return table().io();
+  }
+
+  @Override
+  protected List<DataFile> rewriteDataForTasks(List<CombinedScanTask> combinedScanTasks) {
+    DataSet<CombinedScanTask> dataSet = env.fromCollection(combinedScanTasks).setParallelism(combinedScanTasks.size());

Review comment:
       > Thanks @zhangjun0x01 for your contribution, why do you want to use `DataSet` API? `DataSet` API is being deprecated.
   > Looks like just need a `parallelize` and `map` and `collect`. I think `DataStream`(We can call it BoundedStream) can finish this.
   > What do you think?
   
   Hi,@JingsongLi:
   I look up the api of the DataStream, it seems  no `collect` method

##########
File path: flink/src/main/java/org/apache/iceberg/flink/actions/RewriteDataFilesAction.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.actions;
+
+import java.util.List;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.actions.BaseRewriteDataFilesAction;
+import org.apache.iceberg.flink.source.RowDataRewriter;
+import org.apache.iceberg.io.FileIO;
+
+public class RewriteDataFilesAction extends BaseRewriteDataFilesAction<RewriteDataFilesAction> {
+
+  private ExecutionEnvironment env;
+
+  public RewriteDataFilesAction(ExecutionEnvironment env, Table table) {
+    super(table);
+    this.env = env;
+  }
+
+  @Override
+  protected FileIO fileIO() {
+    return table().io();
+  }
+
+  @Override
+  protected List<DataFile> rewriteDataForTasks(List<CombinedScanTask> combinedScanTasks) {
+    DataSet<CombinedScanTask> dataSet = env.fromCollection(combinedScanTasks).setParallelism(combinedScanTasks.size());

Review comment:
       I updated the pr, use DataStream instead of DataSet , and set the parallelism to StreamExecutionEnvironment.
   
   but setParallelism for  `env.fromCollection(combinedScanTasks)`  does not seem to have an impact on downstream operator,this is my test job graph for use dataset.   https://blog.csdn.net/zhangjun5965/article/details/109469798
   

##########
File path: flink/src/main/java/org/apache/iceberg/flink/actions/RewriteDataFilesAction.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.actions;
+
+import java.util.List;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.actions.BaseRewriteDataFilesAction;
+import org.apache.iceberg.flink.source.RowDataRewriter;
+import org.apache.iceberg.io.FileIO;
+
+public class RewriteDataFilesAction extends BaseRewriteDataFilesAction<RewriteDataFilesAction> {
+
+  private ExecutionEnvironment env;
+
+  public RewriteDataFilesAction(ExecutionEnvironment env, Table table) {
+    super(table);
+    this.env = env;
+  }
+
+  @Override
+  protected FileIO fileIO() {
+    return table().io();
+  }
+
+  @Override
+  protected List<DataFile> rewriteDataForTasks(List<CombinedScanTask> combinedScanTasks) {
+    DataSet<CombinedScanTask> dataSet = env.fromCollection(combinedScanTasks).setParallelism(combinedScanTasks.size());

Review comment:
       If we do not set the parallelism, we can use the default parallelism of flink. If the user wants to modify the parallelism to improve the execution speed, it can be set by the `-p` parameter. However, the user may not know how much parallelism to set,because users may not know how many data files to Rewrite each time. If the setting is too large, it may cause a waste of resources. 
   
   So I think we can  set a max parallelism. If the size of CombinedScanTask list  is less than max parallelism, we use the CombinedScanTask list size as the job parallelism, if the size  is greater than max parallelism,we use the max parallelism which we set.
   What do you think?

##########
File path: flink/src/main/java/org/apache/iceberg/flink/actions/RewriteDataFilesAction.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.actions;
+
+import java.util.List;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.actions.BaseRewriteDataFilesAction;
+import org.apache.iceberg.flink.source.RowDataRewriter;
+import org.apache.iceberg.io.FileIO;
+
+public class RewriteDataFilesAction extends BaseRewriteDataFilesAction<RewriteDataFilesAction> {
+
+  private ExecutionEnvironment env;
+
+  public RewriteDataFilesAction(ExecutionEnvironment env, Table table) {
+    super(table);
+    this.env = env;
+  }
+
+  @Override
+  protected FileIO fileIO() {
+    return table().io();
+  }
+
+  @Override
+  protected List<DataFile> rewriteDataForTasks(List<CombinedScanTask> combinedScanTasks) {
+    DataSet<CombinedScanTask> dataSet = env.fromCollection(combinedScanTasks).setParallelism(combinedScanTasks.size());

Review comment:
       I updated the pr, although we set the parallelism for StreamExecutionEnvironment, I found that the DataStreamUtils.collect operator only uses 1 parallelism, so set parallelism to map operator or StreamExecutionEnvironment seems to have the same effect, but set parallelism to map is better to understand .
   
   I submit an issue (#1660) to  set max parallelism a few days ago, I will open a new pr  for #1660 when this PR is merged




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] JingsongLi commented on a change in pull request #1623: Flink: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
JingsongLi commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r517183685



##########
File path: core/src/main/java/org/apache/iceberg/actions/BaseRewriteDataFilesAction.java
##########
@@ -268,6 +268,23 @@ private void replaceDataFiles(Iterable<DataFile> deletedDataFiles, Iterable<Data
     }
   }
 
+  protected int getParallelism(int parallelism, List<CombinedScanTask> combinedScanTasks) {

Review comment:
       +1 I think we can move options to Flink too.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] zhangjun0x01 commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
zhangjun0x01 commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r507561567



##########
File path: flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.RowDataTaskWriterFactory;
+import org.apache.iceberg.flink.sink.TaskWriterFactory;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.LocationProvider;
+import org.apache.iceberg.io.TaskWriter;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
+
+public class RowDataRewriter implements Serializable {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RowDataRewriter.class);
+
+  private final Schema schema;
+  private final PartitionSpec spec;
+  private final Map<String, String> properties;
+  private final FileFormat format;
+  private final LocationProvider locations;
+  private final String nameMapping;
+  private final FileIO io;
+  private final boolean caseSensitive;
+  private final EncryptionManager encryptionManager;
+  private final Table table;
+
+  private final TaskWriterFactory<RowData> taskWriterFactory;
+
+  public RowDataRewriter(Table table, PartitionSpec spec, boolean caseSensitive,
+                         FileIO io, EncryptionManager encryptionManager) {
+    this.table = table;
+    this.schema = table.schema();
+    this.spec = spec;
+    this.locations = table.locationProvider();
+    this.properties = table.properties();
+    this.io = io;
+    this.encryptionManager = encryptionManager;
+
+    this.caseSensitive = caseSensitive;
+    this.nameMapping = table.properties().get(DEFAULT_NAME_MAPPING);
+
+    String formatString = table.properties().getOrDefault(
+        TableProperties.DEFAULT_FILE_FORMAT, TableProperties.DEFAULT_FILE_FORMAT_DEFAULT);
+    this.format = FileFormat.valueOf(formatString.toUpperCase(Locale.ENGLISH));
+
+    RowType flinkSchema = FlinkSchemaUtil.convert(table.schema());
+    taskWriterFactory = new RowDataTaskWriterFactory(table.schema(),
+        flinkSchema,
+        table.spec(),
+        table.locationProvider(),
+        table.io(),
+        table.encryption(),
+        Long.MAX_VALUE,

Review comment:
       I have read this discussion, but my understanding is that the limit of targetFileSizeBytes is in the reader, not the writer. 
   the writer of the spark is [here](https://github.com/apache/iceberg/blob/master/spark/src/main/java/org/apache/iceberg/spark/source/RowDataRewriter.java#L105)
   
   ```
       TaskWriter<InternalRow> writer;
       if (spec.fields().isEmpty()) {
         writer = new UnpartitionedWriter<>(spec, format, appenderFactory, fileFactory, io.value(), Long.MAX_VALUE);
       } else {
         writer = new SparkPartitionedWriter(spec, format, appenderFactory, fileFactory, io.value(), Long.MAX_VALUE,
             schema, structType);
       }
   ```
   
   I set the targetFileSizeBytes in [RewriteDataFilesActionBase#targetSizeInBytes method ](https://github.com/zhangjun0x01/iceberg/blob/RewriteDataFilesAction/core/src/main/java/org/apache/iceberg/actions/RewriteDataFilesActionBase.java#L200)
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] openinx commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r507541926



##########
File path: flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.RowDataTaskWriterFactory;
+import org.apache.iceberg.flink.sink.TaskWriterFactory;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.LocationProvider;
+import org.apache.iceberg.io.TaskWriter;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
+
+public class RowDataRewriter implements Serializable {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RowDataRewriter.class);
+
+  private final Schema schema;
+  private final PartitionSpec spec;

Review comment:
       Those variables `spec`, `properties`, `locations` are never used in this classes , do we need them here ? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] JingsongLi commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
JingsongLi commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r516439114



##########
File path: flink/src/main/java/org/apache/iceberg/flink/actions/RewriteDataFilesAction.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.actions;
+
+import java.util.List;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.actions.BaseRewriteDataFilesAction;
+import org.apache.iceberg.flink.source.RowDataRewriter;
+import org.apache.iceberg.io.FileIO;
+
+public class RewriteDataFilesAction extends BaseRewriteDataFilesAction<RewriteDataFilesAction> {
+
+  private ExecutionEnvironment env;
+
+  public RewriteDataFilesAction(ExecutionEnvironment env, Table table) {
+    super(table);
+    this.env = env;
+  }
+
+  @Override
+  protected FileIO fileIO() {
+    return table().io();
+  }
+
+  @Override
+  protected List<DataFile> rewriteDataForTasks(List<CombinedScanTask> combinedScanTasks) {
+    DataSet<CombinedScanTask> dataSet = env.fromCollection(combinedScanTasks).setParallelism(combinedScanTasks.size());

Review comment:
       I think `setParallelism` may not works.
   What we want is a `SparkContext.parallelize`, but this `setParallelism` just set parallelism, records still in a single node, downstream operators only have computation on one node.
   Can you check this?

##########
File path: flink/src/main/java/org/apache/iceberg/flink/actions/RewriteDataFilesAction.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.actions;
+
+import java.util.List;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.actions.BaseRewriteDataFilesAction;
+import org.apache.iceberg.flink.source.RowDataRewriter;
+import org.apache.iceberg.io.FileIO;
+
+public class RewriteDataFilesAction extends BaseRewriteDataFilesAction<RewriteDataFilesAction> {
+
+  private ExecutionEnvironment env;
+
+  public RewriteDataFilesAction(ExecutionEnvironment env, Table table) {
+    super(table);
+    this.env = env;
+  }
+
+  @Override
+  protected FileIO fileIO() {
+    return table().io();
+  }
+
+  @Override
+  protected List<DataFile> rewriteDataForTasks(List<CombinedScanTask> combinedScanTasks) {
+    DataSet<CombinedScanTask> dataSet = env.fromCollection(combinedScanTasks).setParallelism(combinedScanTasks.size());

Review comment:
       You can use `DataStreamUtils.collect`

##########
File path: flink/src/main/java/org/apache/iceberg/flink/actions/RewriteDataFilesAction.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.actions;
+
+import java.util.List;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.actions.BaseRewriteDataFilesAction;
+import org.apache.iceberg.flink.source.RowDataRewriter;
+import org.apache.iceberg.io.FileIO;
+
+public class RewriteDataFilesAction extends BaseRewriteDataFilesAction<RewriteDataFilesAction> {
+
+  private ExecutionEnvironment env;
+
+  public RewriteDataFilesAction(ExecutionEnvironment env, Table table) {
+    super(table);
+    this.env = env;
+  }
+
+  @Override
+  protected FileIO fileIO() {
+    return table().io();
+  }
+
+  @Override
+  protected List<DataFile> rewriteDataForTasks(List<CombinedScanTask> combinedScanTasks) {
+    DataSet<CombinedScanTask> dataSet = env.fromCollection(combinedScanTasks).setParallelism(combinedScanTasks.size());

Review comment:
       Thanks for the update and check, I think you are right, the graph is good.

##########
File path: flink/src/main/java/org/apache/iceberg/flink/actions/RewriteDataFilesAction.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.actions;
+
+import java.util.List;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.actions.BaseRewriteDataFilesAction;
+import org.apache.iceberg.flink.source.RowDataRewriter;
+import org.apache.iceberg.io.FileIO;
+
+public class RewriteDataFilesAction extends BaseRewriteDataFilesAction<RewriteDataFilesAction> {
+
+  private ExecutionEnvironment env;
+
+  public RewriteDataFilesAction(ExecutionEnvironment env, Table table) {
+    super(table);
+    this.env = env;
+  }
+
+  @Override
+  protected FileIO fileIO() {
+    return table().io();
+  }
+
+  @Override
+  protected List<DataFile> rewriteDataForTasks(List<CombinedScanTask> combinedScanTasks) {
+    DataSet<CombinedScanTask> dataSet = env.fromCollection(combinedScanTasks).setParallelism(combinedScanTasks.size());

Review comment:
       I think about parallelism again.
   Can we not set parallelism? Flink is a push based model, which is not suitable for taking on too much parallelism.
   So can we just let it as default parallelims?

##########
File path: flink/src/main/java/org/apache/iceberg/flink/actions/RewriteDataFilesAction.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.actions;
+
+import java.util.List;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.actions.BaseRewriteDataFilesAction;
+import org.apache.iceberg.flink.source.RowDataRewriter;
+import org.apache.iceberg.io.FileIO;
+
+public class RewriteDataFilesAction extends BaseRewriteDataFilesAction<RewriteDataFilesAction> {
+
+  private ExecutionEnvironment env;
+
+  public RewriteDataFilesAction(ExecutionEnvironment env, Table table) {
+    super(table);
+    this.env = env;
+  }
+
+  @Override
+  protected FileIO fileIO() {
+    return table().io();
+  }
+
+  @Override
+  protected List<DataFile> rewriteDataForTasks(List<CombinedScanTask> combinedScanTasks) {
+    DataSet<CombinedScanTask> dataSet = env.fromCollection(combinedScanTasks).setParallelism(combinedScanTasks.size());

Review comment:
       Let me explain why we can't set parallelism directly:
   Flink is in pipeline mode by default, which means that all tasks must be started.
   If we want to compact a large table, it may require thousands of parallelism, which leads to the need for a very large cluster.
   
   This is similar to the Hive source parallelism inference: https://ci.apache.org/projects/flink/flink-docs-master/dev/table/hive/hive_read_write.html#source-parallelism-inference
   The value should be used to infer parallelism, but we need a max parallelism.
   
   So I agree with you very much. Let's add a max parallelism.

##########
File path: flink/src/main/java/org/apache/iceberg/flink/actions/RewriteDataFilesAction.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.actions;
+
+import java.util.List;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.actions.BaseRewriteDataFilesAction;
+import org.apache.iceberg.flink.source.RowDataRewriter;
+import org.apache.iceberg.io.FileIO;
+
+public class RewriteDataFilesAction extends BaseRewriteDataFilesAction<RewriteDataFilesAction> {
+
+  private ExecutionEnvironment env;
+
+  public RewriteDataFilesAction(ExecutionEnvironment env, Table table) {
+    super(table);
+    this.env = env;
+  }
+
+  @Override
+  protected FileIO fileIO() {
+    return table().io();
+  }
+
+  @Override
+  protected List<DataFile> rewriteDataForTasks(List<CombinedScanTask> combinedScanTasks) {
+    DataSet<CombinedScanTask> dataSet = env.fromCollection(combinedScanTasks).setParallelism(combinedScanTasks.size());

Review comment:
       BTW, you should set parallelism after `map` instead of modifying env.

##########
File path: flink/src/main/java/org/apache/iceberg/flink/actions/RewriteDataFilesAction.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.actions;
+
+import java.util.List;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.actions.BaseRewriteDataFilesAction;
+import org.apache.iceberg.flink.source.RowDataRewriter;
+import org.apache.iceberg.io.FileIO;
+
+public class RewriteDataFilesAction extends BaseRewriteDataFilesAction<RewriteDataFilesAction> {
+
+  private ExecutionEnvironment env;
+
+  public RewriteDataFilesAction(ExecutionEnvironment env, Table table) {
+    super(table);
+    this.env = env;
+  }
+
+  @Override
+  protected FileIO fileIO() {
+    return table().io();
+  }
+
+  @Override
+  protected List<DataFile> rewriteDataForTasks(List<CombinedScanTask> combinedScanTasks) {
+    DataSet<CombinedScanTask> dataSet = env.fromCollection(combinedScanTasks).setParallelism(combinedScanTasks.size());

Review comment:
       BTW, you should set parallelism for `map` operation instead of modifying env.

##########
File path: flink/src/main/java/org/apache/iceberg/flink/actions/RewriteDataFilesAction.java
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.actions;
+
+import java.util.List;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.actions.BaseRewriteDataFilesAction;
+import org.apache.iceberg.flink.source.RowDataRewriter;
+import org.apache.iceberg.io.FileIO;
+
+public class RewriteDataFilesAction extends BaseRewriteDataFilesAction<RewriteDataFilesAction> {
+
+  private ExecutionEnvironment env;
+
+  public RewriteDataFilesAction(ExecutionEnvironment env, Table table) {
+    super(table);
+    this.env = env;
+  }
+
+  @Override
+  protected FileIO fileIO() {
+    return table().io();
+  }
+
+  @Override
+  protected List<DataFile> rewriteDataForTasks(List<CombinedScanTask> combinedScanTasks) {
+    DataSet<CombinedScanTask> dataSet = env.fromCollection(combinedScanTasks).setParallelism(combinedScanTasks.size());

Review comment:
       > although we set the parallelism for StreamExecutionEnvironment, I found that the DataStreamUtils.collect operator only uses 1 parallelism, so set parallelism to map operator or StreamExecutionEnvironment seems to have the same effect, but set parallelism to map is better to understand.
   
   It is better not to bring extra impact to env, because this env may be reused by users.
   
   > I submit an issue (#1660) to set max parallelism a few days ago, I will open a new pr for #1660 when this PR is merged
   
   Can you modify in this PR? Or just not set parallelism in this PR?
   I think it is not good to bring user cluster risk to master code.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] zhangjun0x01 commented on a change in pull request #1623: Flink: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
zhangjun0x01 commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r517210481



##########
File path: core/src/main/java/org/apache/iceberg/actions/BaseRewriteDataFilesAction.java
##########
@@ -268,6 +268,23 @@ private void replaceDataFiles(Iterable<DataFile> deletedDataFiles, Iterable<Data
     }
   }
 
+  protected int getParallelism(int parallelism, List<CombinedScanTask> combinedScanTasks) {

Review comment:
       should we set the max parallelism for spark  RewriteDataFilesAction? I put the method to BaseRewriteDataFilesAction, because both flink and spark use this method




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] zhangjun0x01 commented on a change in pull request #1623: Flink: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
zhangjun0x01 commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r517075736



##########
File path: flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.stream.Collectors;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.RowDataTaskWriterFactory;
+import org.apache.iceberg.flink.sink.TaskWriterFactory;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.TaskWriter;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
+
+public class RowDataRewriter {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RowDataRewriter.class);
+
+  private final Schema schema;
+  private final FileFormat format;
+  private final String nameMapping;
+  private final FileIO io;
+  private final boolean caseSensitive;
+  private final EncryptionManager encryptionManager;
+  private final TaskWriterFactory<RowData> taskWriterFactory;
+
+  public RowDataRewriter(Table table, boolean caseSensitive, FileIO io, EncryptionManager encryptionManager) {
+    this.schema = table.schema();
+    this.caseSensitive = caseSensitive;
+    this.io = io;
+    this.encryptionManager = encryptionManager;
+    this.nameMapping = PropertyUtil.propertyAsString(table.properties(), DEFAULT_NAME_MAPPING, null);
+
+    String formatString = PropertyUtil.propertyAsString(table.properties(), TableProperties.DEFAULT_FILE_FORMAT,
+        TableProperties.DEFAULT_FILE_FORMAT_DEFAULT);
+    this.format = FileFormat.valueOf(formatString.toUpperCase(Locale.ENGLISH));
+    RowType flinkSchema = FlinkSchemaUtil.convert(table.schema());
+    this.taskWriterFactory = new RowDataTaskWriterFactory(
+        table.schema(),
+        flinkSchema,
+        table.spec(),
+        table.locationProvider(),
+        io,
+        encryptionManager,
+        Long.MAX_VALUE,
+        format,
+        table.properties());
+  }
+
+  public List<DataFile> rewriteDataForTasks(DataStream<CombinedScanTask> dataSet, int parallelism) {

Review comment:
       I updated it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] JingsongLi commented on pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
JingsongLi commented on pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#issuecomment-721012326


   BTW, @zhangjun0x01 It is better to modify PR title to `Flink: Add a rewrite datafile action`


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] openinx commented on a change in pull request #1623: Flink: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r517178412



##########
File path: core/src/main/java/org/apache/iceberg/actions/BaseRewriteDataFilesAction.java
##########
@@ -268,6 +268,23 @@ private void replaceDataFiles(Iterable<DataFile> deletedDataFiles, Iterable<Data
     }
   }
 
+  protected int getParallelism(int parallelism, List<CombinedScanTask> combinedScanTasks) {

Review comment:
       Do we need to pass the whole `List<CombinedScanTask>` here ?  I think we only need the `numOfScanTasks` ? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] zhangjun0x01 commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
zhangjun0x01 commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r507588355



##########
File path: flink/src/main/java/org/apache/iceberg/actions/RewriteDataFilesAction.java
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.actions;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.source.RowDataRewriter;
+import org.apache.iceberg.util.StructLikeWrapper;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class RewriteDataFilesAction extends
+    RewriteDataFilesActionBase<RewriteDataFilesAction, RewriteDataFilesActionResult> {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RewriteDataFilesAction.class);
+
+  private ExecutionEnvironment env = null;
+  private boolean caseSensitive;
+
+  public RewriteDataFilesAction(ExecutionEnvironment env, Table table) {
+    super(table);
+    this.env = env;
+    caseSensitive = Boolean.parseBoolean(table.properties().getOrDefault("case-sensitive", "false"));
+  }
+
+
+  @Override
+  public RewriteDataFilesActionResult execute() {
+    Map<StructLikeWrapper, Collection<FileScanTask>> filteredGroupedTasks = getFilteredGroupedTasks();
+
+    // Nothing to rewrite if there's only one DataFile in each partition.
+    if (filteredGroupedTasks.isEmpty()) {
+      return RewriteDataFilesActionResult.empty();
+    }
+
+    List<DataFile> currentDataFiles = getCurrentDataFiles(filteredGroupedTasks);
+
+    List<CombinedScanTask> combinedScanTasks = getCombinedScanTasks(filteredGroupedTasks);
+
+    DataSet<CombinedScanTask> dataSet = env.fromCollection(combinedScanTasks);
+    env.setParallelism(combinedScanTasks.size());
+
+    RowDataRewriter rowDataRewriter =
+        new RowDataRewriter(getTable(), getTable().spec(), caseSensitive, getTable().io(), getEncryptionManager());
+
+    List<DataFile> addedDataFiles = null;
+    try {
+      addedDataFiles = rowDataRewriter.rewriteDataForTasks(dataSet);
+    } catch (Exception e) {
+      LOG.error("rewrite data files error", e);

Review comment:
       This exception comes from the DataSet#collect method. I looked at this method again. It throws a RuntimeException. I think we also throw a RuntimeException here. What do you think?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] openinx commented on pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
openinx commented on pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#issuecomment-711494157


   Thanks @zhangjun0x01 and @simonsssu  for the work ,  I will take a look today. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] zhangjun0x01 commented on a change in pull request #1623: Flink: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
zhangjun0x01 commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r517250367



##########
File path: core/src/main/java/org/apache/iceberg/actions/BaseRewriteDataFilesAction.java
##########
@@ -268,6 +268,23 @@ private void replaceDataFiles(Iterable<DataFile> deletedDataFiles, Iterable<Data
     }
   }
 
+  protected int getParallelism(int parallelism, List<CombinedScanTask> combinedScanTasks) {

Review comment:
       I moved set max parallelism to flink RewriteDataFilesAction




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] openinx commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r507572845



##########
File path: flink/src/main/java/org/apache/iceberg/actions/RewriteDataFilesAction.java
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.actions;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.flink.source.RowDataRewriter;
+import org.apache.iceberg.util.StructLikeWrapper;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class RewriteDataFilesAction extends
+    RewriteDataFilesActionBase<RewriteDataFilesAction, RewriteDataFilesActionResult> {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RewriteDataFilesAction.class);
+
+  private ExecutionEnvironment env = null;
+  private boolean caseSensitive;
+
+  public RewriteDataFilesAction(ExecutionEnvironment env, Table table) {
+    super(table);
+    this.env = env;
+    caseSensitive = Boolean.parseBoolean(table.properties().getOrDefault("case-sensitive", "false"));
+  }
+
+
+  @Override
+  public RewriteDataFilesActionResult execute() {
+    Map<StructLikeWrapper, Collection<FileScanTask>> filteredGroupedTasks = getFilteredGroupedTasks();
+
+    // Nothing to rewrite if there's only one DataFile in each partition.
+    if (filteredGroupedTasks.isEmpty()) {
+      return RewriteDataFilesActionResult.empty();
+    }
+
+    List<DataFile> currentDataFiles = getCurrentDataFiles(filteredGroupedTasks);
+
+    List<CombinedScanTask> combinedScanTasks = getCombinedScanTasks(filteredGroupedTasks);
+
+    DataSet<CombinedScanTask> dataSet = env.fromCollection(combinedScanTasks);
+    env.setParallelism(combinedScanTasks.size());
+
+    RowDataRewriter rowDataRewriter =
+        new RowDataRewriter(getTable(), getTable().spec(), caseSensitive, getTable().io(), getEncryptionManager());
+
+    List<DataFile> addedDataFiles = null;
+    try {
+      addedDataFiles = rowDataRewriter.rewriteDataForTasks(dataSet);
+    } catch (Exception e) {
+      LOG.error("rewrite data files error", e);

Review comment:
       I think here we'd better to throw the exception to upper layer ?  How about throwing the `UncheckedIOException(e)` ? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] openinx merged pull request #1623: Flink: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
openinx merged pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] openinx commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r507565414



##########
File path: flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.RowDataTaskWriterFactory;
+import org.apache.iceberg.flink.sink.TaskWriterFactory;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.LocationProvider;
+import org.apache.iceberg.io.TaskWriter;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
+
+public class RowDataRewriter implements Serializable {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RowDataRewriter.class);
+
+  private final Schema schema;
+  private final PartitionSpec spec;
+  private final Map<String, String> properties;
+  private final FileFormat format;
+  private final LocationProvider locations;
+  private final String nameMapping;
+  private final FileIO io;
+  private final boolean caseSensitive;
+  private final EncryptionManager encryptionManager;
+  private final Table table;
+
+  private final TaskWriterFactory<RowData> taskWriterFactory;
+
+  public RowDataRewriter(Table table, PartitionSpec spec, boolean caseSensitive,
+                         FileIO io, EncryptionManager encryptionManager) {
+    this.table = table;
+    this.schema = table.schema();
+    this.spec = spec;
+    this.locations = table.locationProvider();
+    this.properties = table.properties();
+    this.io = io;
+    this.encryptionManager = encryptionManager;
+
+    this.caseSensitive = caseSensitive;
+    this.nameMapping = table.properties().get(DEFAULT_NAME_MAPPING);
+
+    String formatString = table.properties().getOrDefault(
+        TableProperties.DEFAULT_FILE_FORMAT, TableProperties.DEFAULT_FILE_FORMAT_DEFAULT);
+    this.format = FileFormat.valueOf(formatString.toUpperCase(Locale.ENGLISH));
+
+    RowType flinkSchema = FlinkSchemaUtil.convert(table.schema());
+    taskWriterFactory = new RowDataTaskWriterFactory(table.schema(),
+        flinkSchema,
+        table.spec(),
+        table.locationProvider(),
+        table.io(),
+        table.encryption(),
+        Long.MAX_VALUE,
+        format,
+        table.properties());
+  }
+
+
+  public List<DataFile> rewriteDataForTasks(DataSet<CombinedScanTask> dataSet) throws Exception {
+
+    RewriteMap map = new RewriteMap(schema, nameMapping, io, caseSensitive,
+        encryptionManager,
+        taskWriterFactory);
+
+    DataSet<List<DataFile>> ds = dataSet.map(map);
+    return ds.collect().stream()
+        .flatMap(Collection::stream)
+        .collect(Collectors.toList());
+  }
+
+  public static class RewriteMap extends RichMapFunction<CombinedScanTask, List<DataFile>> {
+
+    private transient TaskWriter<RowData> writer;
+    private transient int subTaskId;
+    private transient int attemptId;
+
+    private final Schema schema;
+    private final String nameMapping;
+    private final FileIO io;
+    private final boolean caseSensitive;
+    private final EncryptionManager encryptionManager;
+    private final TaskWriterFactory<RowData> taskWriterFactory;
+
+    public RewriteMap(Schema schema, String nameMapping, FileIO io, boolean caseSensitive,
+                      EncryptionManager encryptionManager,
+                      TaskWriterFactory<RowData> taskWriterFactory) {
+      this.schema = schema;
+      this.nameMapping = nameMapping;
+      this.io = io;
+      this.caseSensitive = caseSensitive;
+      this.encryptionManager = encryptionManager;
+      this.taskWriterFactory = taskWriterFactory;
+    }
+
+    @Override
+    public void open(Configuration parameters) {
+      this.subTaskId = getRuntimeContext().getIndexOfThisSubtask();
+      this.attemptId = getRuntimeContext().getAttemptNumber();
+
+      // Initialize the task writer factory.
+      this.taskWriterFactory.initialize(subTaskId, attemptId);
+
+    }
+
+
+    @Override
+    public List<DataFile> map(CombinedScanTask task) throws Exception {
+      // Initialize the task writer.
+      this.writer = taskWriterFactory.create();
+      RowDataIterator iterator =
+          new RowDataIterator(task, io, encryptionManager, schema, schema,
+              nameMapping, caseSensitive);
+      try {
+        while (iterator.hasNext()) {
+          RowData rowData = iterator.next();
+          writer.write(rowData);
+        }
+        iterator.close();
+        writer.close();
+        return Lists.newArrayList(writer.complete()).stream()
+            .collect(Collectors.toList());
+      } catch (Throwable originalThrowable) {
+        try {
+          LOG.error("Aborting task", originalThrowable);
+
+          LOG.error("Aborting commit for  (subTaskId {}, attemptId {})",
+              subTaskId, attemptId);

Review comment:
       nit: Do we need to switch to a new line here ? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] openinx commented on pull request #1623: Flink: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
openinx commented on pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#issuecomment-721596157


   @JingsongLi   Any other concern for this patch ?  


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] zhangjun0x01 commented on a change in pull request #1623: Flink: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
zhangjun0x01 commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r517210481



##########
File path: core/src/main/java/org/apache/iceberg/actions/BaseRewriteDataFilesAction.java
##########
@@ -268,6 +268,23 @@ private void replaceDataFiles(Iterable<DataFile> deletedDataFiles, Iterable<Data
     }
   }
 
+  protected int getParallelism(int parallelism, List<CombinedScanTask> combinedScanTasks) {

Review comment:
       should we set the max parallelism for spark  RewriteDataFilesAction?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] openinx commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r507538851



##########
File path: flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.RowDataTaskWriterFactory;
+import org.apache.iceberg.flink.sink.TaskWriterFactory;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.LocationProvider;
+import org.apache.iceberg.io.TaskWriter;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
+
+public class RowDataRewriter implements Serializable {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RowDataRewriter.class);
+
+  private final Schema schema;
+  private final PartitionSpec spec;
+  private final Map<String, String> properties;
+  private final FileFormat format;
+  private final LocationProvider locations;
+  private final String nameMapping;
+  private final FileIO io;
+  private final boolean caseSensitive;
+  private final EncryptionManager encryptionManager;
+  private final Table table;
+
+  private final TaskWriterFactory<RowData> taskWriterFactory;
+
+  public RowDataRewriter(Table table, PartitionSpec spec, boolean caseSensitive,
+                         FileIO io, EncryptionManager encryptionManager) {
+    this.table = table;
+    this.schema = table.schema();
+    this.spec = spec;
+    this.locations = table.locationProvider();
+    this.properties = table.properties();
+    this.io = io;
+    this.encryptionManager = encryptionManager;
+
+    this.caseSensitive = caseSensitive;
+    this.nameMapping = table.properties().get(DEFAULT_NAME_MAPPING);
+
+    String formatString = table.properties().getOrDefault(
+        TableProperties.DEFAULT_FILE_FORMAT, TableProperties.DEFAULT_FILE_FORMAT_DEFAULT);
+    this.format = FileFormat.valueOf(formatString.toUpperCase(Locale.ENGLISH));
+
+    RowType flinkSchema = FlinkSchemaUtil.convert(table.schema());
+    taskWriterFactory = new RowDataTaskWriterFactory(table.schema(),
+        flinkSchema,
+        table.spec(),
+        table.locationProvider(),
+        table.io(),
+        table.encryption(),
+        Long.MAX_VALUE,
+        format,
+        table.properties());
+  }
+
+
+  public List<DataFile> rewriteDataForTasks(DataSet<CombinedScanTask> dataSet) throws Exception {
+
+    RewriteMap map = new RewriteMap(schema, nameMapping, io, caseSensitive,
+        encryptionManager,
+        taskWriterFactory);
+
+    DataSet<List<DataFile>> ds = dataSet.map(map);
+    return ds.collect().stream()
+        .flatMap(Collection::stream)
+        .collect(Collectors.toList());
+  }
+
+  public static class RewriteMap extends RichMapFunction<CombinedScanTask, List<DataFile>> {
+
+    private transient TaskWriter<RowData> writer;
+    private transient int subTaskId;
+    private transient int attemptId;
+
+    private final Schema schema;
+    private final String nameMapping;
+    private final FileIO io;
+    private final boolean caseSensitive;
+    private final EncryptionManager encryptionManager;
+    private final TaskWriterFactory<RowData> taskWriterFactory;
+
+    public RewriteMap(Schema schema, String nameMapping, FileIO io, boolean caseSensitive,
+                      EncryptionManager encryptionManager,
+                      TaskWriterFactory<RowData> taskWriterFactory) {
+      this.schema = schema;
+      this.nameMapping = nameMapping;
+      this.io = io;
+      this.caseSensitive = caseSensitive;
+      this.encryptionManager = encryptionManager;
+      this.taskWriterFactory = taskWriterFactory;
+    }
+
+    @Override
+    public void open(Configuration parameters) {
+      this.subTaskId = getRuntimeContext().getIndexOfThisSubtask();
+      this.attemptId = getRuntimeContext().getAttemptNumber();
+
+      // Initialize the task writer factory.
+      this.taskWriterFactory.initialize(subTaskId, attemptId);
+
+    }
+
+
+    @Override
+    public List<DataFile> map(CombinedScanTask task) throws Exception {
+      // Initialize the task writer.
+      this.writer = taskWriterFactory.create();
+      RowDataIterator iterator =
+          new RowDataIterator(task, io, encryptionManager, schema, schema,
+              nameMapping, caseSensitive);
+      try {
+        while (iterator.hasNext()) {
+          RowData rowData = iterator.next();
+          writer.write(rowData);
+        }
+        iterator.close();
+        writer.close();

Review comment:
       The following `writer.complete()` will close firstly and then return the completed data file list, pls see the javadoc in `TaskWriter`: 
   
   ```java
     /**
      * Close the writer and get the completed data files.
      *
      * @return the completed data files of this task writer.
      */
     DataFile[] complete() throws IOException;
   ```
   So here we don't have to call `writer.close()` now.
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] openinx commented on a change in pull request #1623: Flink: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r517180085



##########
File path: core/src/main/java/org/apache/iceberg/actions/BaseRewriteDataFilesAction.java
##########
@@ -268,6 +268,23 @@ private void replaceDataFiles(Iterable<DataFile> deletedDataFiles, Iterable<Data
     }
   }
 
+  protected int getParallelism(int parallelism, List<CombinedScanTask> combinedScanTasks) {

Review comment:
       btw,  it seems to be a method which was designed for flink, Better to move it to flink implementation class ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] openinx commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r507587642



##########
File path: flink/src/main/java/org/apache/iceberg/flink/source/RowDataRewriter.java
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.encryption.EncryptionManager;
+import org.apache.iceberg.flink.FlinkSchemaUtil;
+import org.apache.iceberg.flink.sink.RowDataTaskWriterFactory;
+import org.apache.iceberg.flink.sink.TaskWriterFactory;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.LocationProvider;
+import org.apache.iceberg.io.TaskWriter;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
+
+public class RowDataRewriter implements Serializable {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RowDataRewriter.class);
+
+  private final Schema schema;
+  private final PartitionSpec spec;
+  private final Map<String, String> properties;
+  private final FileFormat format;
+  private final LocationProvider locations;
+  private final String nameMapping;
+  private final FileIO io;
+  private final boolean caseSensitive;
+  private final EncryptionManager encryptionManager;
+  private final Table table;
+
+  private final TaskWriterFactory<RowData> taskWriterFactory;
+
+  public RowDataRewriter(Table table, PartitionSpec spec, boolean caseSensitive,
+                         FileIO io, EncryptionManager encryptionManager) {
+    this.table = table;
+    this.schema = table.schema();
+    this.spec = spec;
+    this.locations = table.locationProvider();
+    this.properties = table.properties();
+    this.io = io;
+    this.encryptionManager = encryptionManager;
+
+    this.caseSensitive = caseSensitive;
+    this.nameMapping = table.properties().get(DEFAULT_NAME_MAPPING);
+
+    String formatString = table.properties().getOrDefault(
+        TableProperties.DEFAULT_FILE_FORMAT, TableProperties.DEFAULT_FILE_FORMAT_DEFAULT);
+    this.format = FileFormat.valueOf(formatString.toUpperCase(Locale.ENGLISH));
+
+    RowType flinkSchema = FlinkSchemaUtil.convert(table.schema());
+    taskWriterFactory = new RowDataTaskWriterFactory(table.schema(),
+        flinkSchema,
+        table.spec(),
+        table.locationProvider(),
+        table.io(),
+        table.encryption(),
+        Long.MAX_VALUE,

Review comment:
       The conclusion should be [here](https://github.com/apache/iceberg/pull/1083/files#r439180961).  OK,  we only have to control the read split size and ignore the write target file size. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] simonsssu commented on a change in pull request #1623: Add a rewrite datafile action for flink

Posted by GitBox <gi...@apache.org>.
simonsssu commented on a change in pull request #1623:
URL: https://github.com/apache/iceberg/pull/1623#discussion_r508221855



##########
File path: flink/src/main/java/org/apache/iceberg/flink/TablePropertiesUtil.java
##########
@@ -0,0 +1,38 @@
+/*

Review comment:
       Do we really need this class ? Why not use org.apache.iceberg.util.PropertyUtil instead ? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org