You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/11/16 02:26:56 UTC
[GitHub] [iceberg] stevenzwu opened a new pull request #3501: Flink: FLIP-27 Iceberg source split
stevenzwu opened a new pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501
This is against v1.13 only. Will port to v1.14 when it is ready. we will skip v1.12 for FLIP-27 source
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r766912145
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedBytesCache;
+
+ private IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedBytesCache() {
+ return serializedBytesCache;
+ }
+
+ void serializedBytesCache(byte[] cachedBytes) {
+ this.serializedBytesCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ IcebergSourceSplit split = (IcebergSourceSplit) o;
+ return Objects.equal(splitId(), split.splitId()) &&
+ Objects.equal(position, split.position());
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hashCode(splitId());
+ }
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .add("position", position)
+ .toString();
+ }
+
+ private String toString(Collection<FileScanTask> files) {
+ return Iterables.toString(files.stream().map(fileScanTask ->
+ MoreObjects.toStringHelper(fileScanTask)
+ .add("file", fileScanTask.file() != null ?
Review comment:
I think we should fix the mock instead of adding logic to the real code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r766919061
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/Position.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+/**
+ * A mutable class that defines the read position
+ * <ul>
+ * <li>file offset in the list of files in a {@link CombinedScanTask}</li>
+ * <li>record offset within a file</li>
+ * </ul>
+ */
+@Internal
+public class Position implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ private int fileOffset;
+ private long recordOffset;
+
+ public Position(int fileOffset, long recordOffset) {
+ this.fileOffset = fileOffset;
+ this.recordOffset = recordOffset;
+ }
+
+ void advanceFile() {
+ this.fileOffset += 1;
+ this.recordOffset = 0L;
+ }
+
+ void advanceRecord() {
+ this.recordOffset += 1L;
+ }
+
+ public void update(int newFileOffset, long newRecordOffset) {
+ this.fileOffset = newFileOffset;
+ this.recordOffset = newRecordOffset;
+ }
+
+ public int fileOffset() {
+ return fileOffset;
+ }
+
+ public long recordOffset() {
+ return recordOffset;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ Position that = (Position) o;
+ return Objects.equals(fileOffset, that.fileOffset) &&
+ Objects.equals(recordOffset, that.recordOffset);
Review comment:
I tend to prefer `Object.equals` as it immediately does `==` check anyway, and then I don't have to think about it in the future.
If the project style prefers `==` where possible, then by all means go with that.
Object.equals code per OpenJDK 8.
```java
public static boolean equals(Object a, Object b) {
return (a == b) || (a != null && a.equals(b));
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r766910663
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedBytesCache;
+
+ private IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
Review comment:
As position advances, this needs to invalidate the cached bytes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r771992456
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedBytesCache;
+
+ private IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedBytesCache() {
+ return serializedBytesCache;
+ }
+
+ void serializedBytesCache(byte[] cachedBytes) {
+ this.serializedBytesCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ IcebergSourceSplit split = (IcebergSourceSplit) o;
+ return Objects.equal(splitId(), split.splitId()) &&
+ Objects.equal(position, split.position());
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hashCode(splitId());
+ }
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .add("position", position)
+ .toString();
+ }
+
+ private String toString(Collection<FileScanTask> files) {
+ return Iterables.toString(files.stream().map(fileScanTask ->
+ MoreObjects.toStringHelper(fileScanTask)
+ .add("file", fileScanTask.file() != null ?
Review comment:
make sense. will update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r766894812
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/FlinkSplitPlanner.java
##########
@@ -22,33 +22,58 @@
import java.io.IOException;
import java.io.UncheckedIOException;
import java.util.List;
+import org.apache.flink.annotation.Internal;
import org.apache.iceberg.CombinedScanTask;
import org.apache.iceberg.Table;
import org.apache.iceberg.TableProperties;
import org.apache.iceberg.TableScan;
import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.flink.source.split.IcebergSourceSplit;
import org.apache.iceberg.io.CloseableIterable;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-class FlinkSplitGenerator {
- private FlinkSplitGenerator() {
+@Internal
+public class FlinkSplitPlanner {
+ private FlinkSplitPlanner() {
}
- static FlinkInputSplit[] createInputSplits(Table table, ScanContext context) {
- List<CombinedScanTask> tasks = tasks(table, context);
- FlinkInputSplit[] splits = new FlinkInputSplit[tasks.size()];
- for (int i = 0; i < tasks.size(); i++) {
- splits[i] = new FlinkInputSplit(i, tasks.get(i));
+ static FlinkInputSplit[] planInputSplits(Table table, ScanContext context) {
+ try (CloseableIterable<CombinedScanTask> tasksIterable = planTasks(table, context)) {
+ List<CombinedScanTask> tasks = Lists.newArrayList(tasksIterable);
+ FlinkInputSplit[] splits = new FlinkInputSplit[tasks.size()];
+ for (int i = 0; i < tasks.size(); i++) {
+ splits[i] = new FlinkInputSplit(i, tasks.get(i));
+ }
+ return splits;
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to process tasks iterable", e);
+ }
+ }
+
+ /**
+ * This returns splits for the FLIP-27 source
+ */
+ public static List<IcebergSourceSplit> planIcebergSourceSplits(
+ Table table, ScanContext context) {
+ try (CloseableIterable<CombinedScanTask> tasksIterable = planTasks(table, context)) {
+ List<IcebergSourceSplit> splits = Lists.newArrayList();
+ tasksIterable.forEach(task -> splits.add(IcebergSourceSplit.fromCombinedScanTask(task)));
+ return splits;
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to process task iterable: ", e);
}
- return splits;
}
- private static List<CombinedScanTask> tasks(Table table, ScanContext context) {
+ static CloseableIterable<CombinedScanTask> planTasks(Table table, ScanContext context) {
TableScan scan = table
.newScan()
.caseSensitive(context.caseSensitive())
.project(context.project());
+ if (context.includeColumnStats()) {
+ scan = scan.includeColumnStats();
+ }
Review comment:
I think this is a reasonable addition. I think the motivation is to change the file in just this PR and not in the others that are part of FLIP-27.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r766898160
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/Position.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+/**
+ * A mutable class that defines the read position
+ * <ul>
+ * <li>file offset in the list of files in a {@link CombinedScanTask}</li>
+ * <li>record offset within a file</li>
+ * </ul>
+ */
+@Internal
+public class Position implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ private int fileOffset;
+ private long recordOffset;
+
+ public Position(int fileOffset, long recordOffset) {
+ this.fileOffset = fileOffset;
+ this.recordOffset = recordOffset;
+ }
+
+ void advanceFile() {
+ this.fileOffset += 1;
+ this.recordOffset = 0L;
+ }
+
+ void advanceRecord() {
+ this.recordOffset += 1L;
+ }
+
+ public void update(int newFileOffset, long newRecordOffset) {
Review comment:
Seems more like a `set` method to me since it directly sets the internal counters rather than advancing them using an amount.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r766903328
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java
##########
@@ -34,7 +34,7 @@
/**
* Context object with optional arguments for a Flink Scan.
*/
-class ScanContext implements Serializable {
+public class ScanContext implements Serializable {
Review comment:
I think we should just have the minimal set of changes to this file, rather than trying to fix the duplication between this and `FlinkSource.Builder`. Also, we should probably add `@Internal` because this is becoming public.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplitSerializer.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.IOException;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.util.InstantiationUtil;
+
+/**
+ * TODO: use Java serialization for now.
+ * Will switch to more stable serializer from
+ * <a href="https://github.com/apache/iceberg/issues/1698">issue-1698</a>.
+ */
+@Internal
+public class IcebergSourceSplitSerializer implements SimpleVersionedSerializer<IcebergSourceSplit> {
+ public static final IcebergSourceSplitSerializer INSTANCE = new IcebergSourceSplitSerializer();
+ private static final int VERSION = 1;
+
+ @Override
+ public int getVersion() {
+ return VERSION;
+ }
+
+ @Override
+ public byte[] serialize(IcebergSourceSplit split) throws IOException {
+ if (split.serializedBytesCache() == null) {
+ byte[] result = serializeV1(split);
+ split.serializedBytesCache(result);
Review comment:
Can we move serialization inside the split? We could then call `split.serializeV1()` and it could cache internally.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplitSerializer.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.IOException;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.util.InstantiationUtil;
+
+/**
+ * TODO: use Java serialization for now.
+ * Will switch to more stable serializer from
+ * <a href="https://github.com/apache/iceberg/issues/1698">issue-1698</a>.
+ */
+@Internal
+public class IcebergSourceSplitSerializer implements SimpleVersionedSerializer<IcebergSourceSplit> {
+ public static final IcebergSourceSplitSerializer INSTANCE = new IcebergSourceSplitSerializer();
+ private static final int VERSION = 1;
+
+ @Override
+ public int getVersion() {
+ return VERSION;
+ }
+
+ @Override
+ public byte[] serialize(IcebergSourceSplit split) throws IOException {
+ if (split.serializedBytesCache() == null) {
+ byte[] result = serializeV1(split);
+ split.serializedBytesCache(result);
+ }
+ return split.serializedBytesCache();
+ }
+
+ @Override
+ public IcebergSourceSplit deserialize(int version, byte[] serialized) throws IOException {
+ switch (version) {
+ case 1:
+ return deserializeV1(serialized);
+ default:
+ throw new IOException(String.format("Failed to deserialize IcebergSourceSplit. " +
+ "Encountered unsupported version: %d. Supported version are [1]", version));
+ }
+ }
+
+ @VisibleForTesting
+ byte[] serializeV1(IcebergSourceSplit split) throws IOException {
+ return InstantiationUtil.serializeObject(split);
+ }
+
+ @VisibleForTesting
+ IcebergSourceSplit deserializeV1(byte[] serialized) throws IOException {
+ try {
+ return InstantiationUtil.deserializeObject(serialized, getClass().getClassLoader());
Review comment:
Should the `ClassLoader` be `IcebergSourceSplit.class.getClassLoader()`?
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedBytesCache;
+
+ private IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
Review comment:
As position advances, this needs to invalidate the cached bytes.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedBytesCache;
+
+ private IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedBytesCache() {
+ return serializedBytesCache;
+ }
+
+ void serializedBytesCache(byte[] cachedBytes) {
+ this.serializedBytesCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
Review comment:
Invalidate serialized bytes here.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedBytesCache;
+
+ private IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedBytesCache() {
+ return serializedBytesCache;
+ }
+
+ void serializedBytesCache(byte[] cachedBytes) {
+ this.serializedBytesCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ IcebergSourceSplit split = (IcebergSourceSplit) o;
+ return Objects.equal(splitId(), split.splitId()) &&
+ Objects.equal(position, split.position());
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hashCode(splitId());
+ }
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .add("position", position)
+ .toString();
+ }
+
+ private String toString(Collection<FileScanTask> files) {
+ return Iterables.toString(files.stream().map(fileScanTask ->
+ MoreObjects.toStringHelper(fileScanTask)
+ .add("file", fileScanTask.file() != null ?
Review comment:
I think we should fix the mock instead of adding logic to the real code.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedBytesCache;
+
+ private IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedBytesCache() {
+ return serializedBytesCache;
+ }
+
+ void serializedBytesCache(byte[] cachedBytes) {
+ this.serializedBytesCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ IcebergSourceSplit split = (IcebergSourceSplit) o;
+ return Objects.equal(splitId(), split.splitId()) &&
+ Objects.equal(position, split.position());
Review comment:
The `equals` and `hashCode` implementations are not consistent because `equals` uses the position, but `hashCode` ignores it.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedBytesCache;
+
+ private IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedBytesCache() {
+ return serializedBytesCache;
+ }
+
+ void serializedBytesCache(byte[] cachedBytes) {
+ this.serializedBytesCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ IcebergSourceSplit split = (IcebergSourceSplit) o;
+ return Objects.equal(splitId(), split.splitId()) &&
+ Objects.equal(position, split.position());
Review comment:
Seems like you might want to just remove this and `hashCode` and use object identity.
##########
File path: flink/v1.13/flink/src/test/java/org/apache/iceberg/flink/source/split/SplitHelpers.java
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import org.apache.iceberg.BaseCombinedScanTask;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MockFileScanTask;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericAppenderHelper;
+import org.apache.iceberg.data.RandomGenericData;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.flink.TestFixtures;
+import org.apache.iceberg.flink.source.FlinkSplitPlanner;
+import org.apache.iceberg.flink.source.ScanContext;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.Assert;
+import org.junit.rules.TemporaryFolder;
+
+public class SplitHelpers {
+
+ private static final AtomicLong splitLengthIncrement = new AtomicLong();
+
+ private SplitHelpers() {
+
Review comment:
Nit: no need for a newline here.
##########
File path: flink/v1.13/flink/src/test/java/org/apache/iceberg/flink/source/split/SplitHelpers.java
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import org.apache.iceberg.BaseCombinedScanTask;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MockFileScanTask;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericAppenderHelper;
+import org.apache.iceberg.data.RandomGenericData;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.flink.TestFixtures;
+import org.apache.iceberg.flink.source.FlinkSplitPlanner;
+import org.apache.iceberg.flink.source.ScanContext;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.Assert;
+import org.junit.rules.TemporaryFolder;
+
+public class SplitHelpers {
+
+ private static final AtomicLong splitLengthIncrement = new AtomicLong();
+
+ private SplitHelpers() {
+
+ }
+
+ public static List<IcebergSourceSplit> createMockedSplits(int splitCount) {
+ List<IcebergSourceSplit> splits = new ArrayList<>();
+ for (int i = 0; i < splitCount; ++i) {
+ // make sure each task has a different length,
+ // as it is part of the splitId calculation.
+ // This way, we can make sure all generated splits have different splitIds
+ FileScanTask fileScanTask = new MockFileScanTask(1024 + splitLengthIncrement.incrementAndGet());
+ CombinedScanTask combinedScanTask = new BaseCombinedScanTask(fileScanTask);
+ splits.add(IcebergSourceSplit.fromCombinedScanTask(combinedScanTask));
+ }
+ return splits;
+ }
+
+ /**
+ * Unlike {@link SplitHelpers#createMockedSplits(int)} above, this is to generate some
+ * realistic IcebergSourceSplit with actual file paths.
+ *
+ * Actual data files are already deleted before return. Caller shouldn't attempt to read the
+ * data files. They are intended for serializer or enumerator unit test (without actual reading).
+ */
+ public static List<IcebergSourceSplit> createFileSplits(
+ TemporaryFolder temporaryFolder, int fileCount, int filesPerSplit) throws Exception {
+ File warehouseFile = temporaryFolder.newFolder();
+ Assert.assertTrue(warehouseFile.delete());
+ String warehouse = "file:" + warehouseFile;
+ org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
Review comment:
Can we remove the fully-qualified path?
##########
File path: flink/v1.13/flink/src/test/java/org/apache/iceberg/flink/source/split/SplitHelpers.java
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import org.apache.iceberg.BaseCombinedScanTask;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MockFileScanTask;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericAppenderHelper;
+import org.apache.iceberg.data.RandomGenericData;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.flink.TestFixtures;
+import org.apache.iceberg.flink.source.FlinkSplitPlanner;
+import org.apache.iceberg.flink.source.ScanContext;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.Assert;
+import org.junit.rules.TemporaryFolder;
+
+public class SplitHelpers {
+
+ private static final AtomicLong splitLengthIncrement = new AtomicLong();
+
+ private SplitHelpers() {
+
+ }
+
+ public static List<IcebergSourceSplit> createMockedSplits(int splitCount) {
+ List<IcebergSourceSplit> splits = new ArrayList<>();
+ for (int i = 0; i < splitCount; ++i) {
+ // make sure each task has a different length,
+ // as it is part of the splitId calculation.
+ // This way, we can make sure all generated splits have different splitIds
+ FileScanTask fileScanTask = new MockFileScanTask(1024 + splitLengthIncrement.incrementAndGet());
+ CombinedScanTask combinedScanTask = new BaseCombinedScanTask(fileScanTask);
+ splits.add(IcebergSourceSplit.fromCombinedScanTask(combinedScanTask));
+ }
+ return splits;
+ }
+
+ /**
+ * Unlike {@link SplitHelpers#createMockedSplits(int)} above, this is to generate some
+ * realistic IcebergSourceSplit with actual file paths.
+ *
+ * Actual data files are already deleted before return. Caller shouldn't attempt to read the
+ * data files. They are intended for serializer or enumerator unit test (without actual reading).
+ */
+ public static List<IcebergSourceSplit> createFileSplits(
+ TemporaryFolder temporaryFolder, int fileCount, int filesPerSplit) throws Exception {
Review comment:
Is there a better name that shows that this is creating a table, writing data, and then planning a scan?
##########
File path: flink/v1.13/flink/src/test/java/org/apache/iceberg/flink/source/split/TestIcebergSourceSplitSerializer.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.junit.Assert;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+public class TestIcebergSourceSplitSerializer {
Review comment:
Probably want a test for calling `advance` on the position and then serializing the split.
##########
File path: flink/v1.13/flink/src/test/java/org/apache/iceberg/flink/source/split/TestIcebergSourceSplitSerializer.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.junit.Assert;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+public class TestIcebergSourceSplitSerializer {
+
+ @ClassRule
+ public static final TemporaryFolder TEMPORARY_FOLDER = new TemporaryFolder();
+
+ private final IcebergSourceSplitSerializer serializer = IcebergSourceSplitSerializer.INSTANCE;
+
+ @Test
+ public void testLatestVersion() throws Exception {
+ serializeAndDeserialize(1, 1);
+ serializeAndDeserialize(10, 2);
+ }
+
+ private void serializeAndDeserialize(int splitCount, int filesPerSplit) throws Exception {
+ List<IcebergSourceSplit> splits = SplitHelpers.createFileSplits(TEMPORARY_FOLDER, splitCount, filesPerSplit);
+ for (IcebergSourceSplit split : splits) {
+ byte[] result = serializer.serialize(split);
+ IcebergSourceSplit deserialized = serializer.deserialize(serializer.getVersion(), result);
+ Assert.assertEquals(split, deserialized);
+
+ byte[] cachedResult = serializer.serialize(split);
+ Assert.assertSame(result, cachedResult);
+ IcebergSourceSplit deserialized2 = serializer.deserialize(serializer.getVersion(), cachedResult);
+ Assert.assertEquals(split, deserialized2);
+ }
+ }
+
+ @Test
+ public void testV1() throws Exception {
+ serializeAndDeserializeV1(1, 1);
+ serializeAndDeserializeV1(10, 2);
+ }
+
+ private void serializeAndDeserializeV1(int splitCount, int filesPerSplit) throws Exception {
+ List<IcebergSourceSplit> splits = SplitHelpers.createFileSplits(TEMPORARY_FOLDER, splitCount, filesPerSplit);
+ for (IcebergSourceSplit split : splits) {
+ byte[] result = serializer.serializeV1(split);
+ IcebergSourceSplit deserialized = serializer.deserializeV1(result);
+ Assert.assertEquals(split, deserialized);
+ }
+ }
+
+ @Test
+ public void testCheckpointedPosition() throws Exception {
+ AtomicInteger index = new AtomicInteger();
+ List<IcebergSourceSplit> splits = SplitHelpers.createFileSplits(TEMPORARY_FOLDER, 10, 2).stream()
+ .map(split -> {
+ IcebergSourceSplit result;
+ if (index.get() % 2 == 0) {
+ result = IcebergSourceSplit.fromCombinedScanTask(split.task(), index.get(), index.get());
Review comment:
Before this, call `serializer.serialize(...)` to test that serialization is correct after changing the position.
Some of these should also call `advanceFile()` and `advanceRow()` as well.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/Position.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+/**
+ * A mutable class that defines the read position
+ * <ul>
+ * <li>file offset in the list of files in a {@link CombinedScanTask}</li>
+ * <li>record offset within a file</li>
+ * </ul>
+ */
+@Internal
+public class Position implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ private int fileOffset;
+ private long recordOffset;
+
+ public Position(int fileOffset, long recordOffset) {
+ this.fileOffset = fileOffset;
+ this.recordOffset = recordOffset;
+ }
+
+ void advanceFile() {
+ this.fileOffset += 1;
+ this.recordOffset = 0L;
+ }
+
+ void advanceRecord() {
+ this.recordOffset += 1L;
+ }
+
+ public void update(int newFileOffset, long newRecordOffset) {
+ this.fileOffset = newFileOffset;
+ this.recordOffset = newRecordOffset;
+ }
+
+ public int fileOffset() {
+ return fileOffset;
+ }
+
+ public long recordOffset() {
+ return recordOffset;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ Position that = (Position) o;
+ return Objects.equals(fileOffset, that.fileOffset) &&
+ Objects.equals(recordOffset, that.recordOffset);
Review comment:
These are primitives, so calling `equals(Object, Object)` will box both values just to do a more expensive check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx closed pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
openinx closed pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#issuecomment-1008348143
Looks good now. Thanks, @stevenzwu!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r745553502
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/Position.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
Review comment:
Published PR https://github.com/apache/iceberg/pull/3509 for this comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r745401341
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/Position.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
Review comment:
The class `org.apache.iceberg.flink.Position` is not put in the correct package `org.apache.iceberg.flink.source` ?
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedFormCache;
+
+ public IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedFormCache() {
+ return serializedFormCache;
+ }
+
+ void serializedFormCache(byte[] cachedBytes) {
+ this.serializedFormCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ IcebergSourceSplit split = (IcebergSourceSplit) o;
+ return Objects.equal(splitId(), split.splitId()) &&
+ Objects.equal(position, split.position());
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hashCode(splitId());
Review comment:
Using the value of `toString()` to calculate the hasCode is not a good practice because if some `FileScanTask` implementation did not implement the correct `toString()`, it will use the instance's hashCode as its `toString()` value. That leads to the meaningless of comparing two `IcebergSourceSplit`.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/Position.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
Review comment:
When I build the module by using commaned`./gradlew iceberg-flink:iceberg-flink-1.13:build -x test`. it will report the error message:
```
> Task :iceberg-flink:iceberg-flink-1.13:checkstyleMain
[ant:checkstyle] [ERROR] /Users/openinx/software/apache-iceberg/flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/Position.java:20:1: Package name is not same as directory. [PackageDeclaration]
> Task :iceberg-flink:iceberg-flink-1.13:checkstyleMain FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':iceberg-flink:iceberg-flink-1.13:checkstyleMain'.
> Checkstyle rule violations were found. See the report at: file:///Users/openinx/software/apache-iceberg/flink/v1.13/flink/build/reports/checkstyle/main.html
Checkstyle files with violations: 1
Checkstyle violations by severity: [error:1]
```
It's strange that the apache iceberg travis CI did not report this issue. I think I need to take a look what's wrong.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedFormCache;
+
+ public IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedFormCache() {
+ return serializedFormCache;
+ }
+
+ void serializedFormCache(byte[] cachedBytes) {
+ this.serializedFormCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ IcebergSourceSplit split = (IcebergSourceSplit) o;
+ return Objects.equal(splitId(), split.splitId()) &&
+ Objects.equal(position, split.position());
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hashCode(splitId());
+ }
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .add("position", position)
+ .toString();
+ }
+
+ private String toString(Collection<FileScanTask> files) {
+ return Iterables.toString(files.stream().map(fileScanTask ->
+ MoreObjects.toStringHelper(fileScanTask)
+ .add("file", fileScanTask.file() != null ?
+ fileScanTask.file().path().toString() :
+ "NoFile")
Review comment:
I think the `FileScanTask#file()` is guaranteed to provide a non-null value, otherwise the `start()` and `length()` won't have any meaning.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedFormCache;
+
+ public IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedFormCache() {
+ return serializedFormCache;
+ }
+
+ void serializedFormCache(byte[] cachedBytes) {
+ this.serializedFormCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ IcebergSourceSplit split = (IcebergSourceSplit) o;
+ return Objects.equal(splitId(), split.splitId()) &&
+ Objects.equal(position, split.position());
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hashCode(splitId());
Review comment:
As we have the cached `serializedFormCache`, why not use the lazy approach to get the serialized bytes and calculate its byte array's hashCode ?
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedFormCache;
+
+ public IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedFormCache() {
+ return serializedFormCache;
+ }
+
+ void serializedFormCache(byte[] cachedBytes) {
+ this.serializedFormCache = cachedBytes;
+ }
Review comment:
Why not use the lazily approach to serialize the split which is similar to the Schema. https://github.com/apache/iceberg/blob/1d8164383baf0e343ade13b21ed0e4d4d02c56cd/api/src/main/java/org/apache/iceberg/Schema.java#L137
Setting the cachedBytes looks a bit strange for me.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/Position.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
Review comment:
After read this [PR](https://github.com/apache/iceberg/pull/685), I think it's reasonable to disable the baseline check when running the unit tests in local IDE, but it does not make sense to disable it in github travis because it will lead us to merging the PR that introduced basic checkstyle issues.
FYI @rdblue
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplitStatus.java
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+public enum IcebergSourceSplitStatus {
Review comment:
Is this class related to this source split ?
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/Position.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
Review comment:
When I copied the command from flink-ci.yaml and run it in command line, it work fine without reporting any errors :
```bash
./gradlew -DsparkVersions= -DhiveVersions= -DflinkVersions=1.13 :iceberg-flink:iceberg-flink-1.13:check :iceberg-flink:iceberg-flink-1.13-runtime:check -Pquick=true -x javadoc -x test
```
But once we removed the `-Pquick=true`, it will detect the checkstyle issue:
```
./gradlew -DsparkVersions= -DhiveVersions= -DflinkVersions=1.13 :iceberg-flink:iceberg-flink-1.13:check :iceberg-flink:iceberg-flink-1.13-runtime:check -x javadoc -x test
> Task :iceberg-flink:iceberg-flink-1.13:checkstyleMain
[ant:checkstyle] [ERROR] /Users/openinx/software/apache-iceberg/flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/Position.java:20:1: Package name is not same as directory. [PackageDeclaration]
> Task :iceberg-flink:iceberg-flink-1.13:checkstyleMain FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':iceberg-flink:iceberg-flink-1.13:checkstyleMain'.
> Checkstyle rule violations were found. See the report at: file:///Users/openinx/software/apache-iceberg/flink/v1.13/flink/build/reports/checkstyle/main.html
Checkstyle files with violations: 1
Checkstyle violations by severity: [error:1]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r746194600
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
Review comment:
Should we set `serialVersionUID` in case users refactor their DAGs?
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplitSerializer.java
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.IOException;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.util.InstantiationUtil;
+
+/**
+ * TODO: use Java serialization for now.
+ * Will switch to more stable serializer from
+ * <a href="https://github.com/apache/iceberg/issues/1698">issue-1698</a>.
+ */
+@Internal
+public class IcebergSourceSplitSerializer implements SimpleVersionedSerializer<IcebergSourceSplit> {
+ public static final IcebergSourceSplitSerializer INSTANCE = new IcebergSourceSplitSerializer();
+ private static final int VERSION = 1;
+
+ @Override
+ public int getVersion() {
+ return VERSION;
+ }
+
+ @Override
+ public byte[] serialize(IcebergSourceSplit split) throws IOException {
+ if (split.serializedFormCache() == null) {
+ final byte[] result = serializeV1(split);
+ split.serializedFormCache(result);
+ }
+ return split.serializedFormCache();
+ }
+
+ @Override
+ public IcebergSourceSplit deserialize(int version, byte[] serialized) throws IOException {
+ switch (version) {
+ case 1:
+ return deserializeV1(serialized);
+ default:
+ throw new IOException("Unknown version: " + version);
Review comment:
Nit: It might help to mention the highest known version (assuming that we'll use monotonically increasing versioning). Or at the least, since most users won't be aware of `SimpleVersionedSerializer`, mentioning what the unknown version means (seems like this would happen in theory just with data written by a newer library version or maybe corrupted data / some kind of bug).
Something like `Failed to deserialize IcebergSourceSplit. Encountered unknown version $version. The maximum version that can be handled is ${currentVersion}`.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplitSerializer.java
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.IOException;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.util.InstantiationUtil;
+
+/**
+ * TODO: use Java serialization for now.
+ * Will switch to more stable serializer from
+ * <a href="https://github.com/apache/iceberg/issues/1698">issue-1698</a>.
+ */
+@Internal
+public class IcebergSourceSplitSerializer implements SimpleVersionedSerializer<IcebergSourceSplit> {
Review comment:
Will this get used to serialize / deserialize the splits across task boundaries, or just for checkpoints?
The comment in `IcebergSourceSplit` for `serializedFormCache` field mentions checkpoints, but curious about crossing task boundaries etc.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedFormCache;
+
+ public IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedFormCache() {
+ return serializedFormCache;
+ }
+
+ void serializedFormCache(byte[] cachedBytes) {
+ this.serializedFormCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ IcebergSourceSplit split = (IcebergSourceSplit) o;
+ return Objects.equal(splitId(), split.splitId()) &&
+ Objects.equal(position, split.position());
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hashCode(splitId());
Review comment:
I don't think that we have a guarantee that `serializedFormCache` will be set unless the split has in fact been serialized via `IcebergSourceSplitSerializer`.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedFormCache;
Review comment:
Nit: This is a great idea.
However, the name is a little confusing for me. Maybe `lzySerializedBytes` or something (sort of like @openinx's comment)? Just a nit as I don't want to bike shed on the name. Clever idea overall.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedFormCache;
+
+ public IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedFormCache() {
+ return serializedFormCache;
+ }
+
+ void serializedFormCache(byte[] cachedBytes) {
+ this.serializedFormCache = cachedBytes;
+ }
Review comment:
Correct me if I'm wrong, but it seems like it is lazy and only set via the first call to `IcebergSourceSplitSerializer`.
Maybe the name is what looks a bit odd to you? I had to read it a few times myself, but I'm not as quick at Flink stuff as you two (or in general).
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplitSerializer.java
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.IOException;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.util.InstantiationUtil;
+
+/**
+ * TODO: use Java serialization for now.
+ * Will switch to more stable serializer from
+ * <a href="https://github.com/apache/iceberg/issues/1698">issue-1698</a>.
+ */
+@Internal
+public class IcebergSourceSplitSerializer implements SimpleVersionedSerializer<IcebergSourceSplit> {
Review comment:
Question for my own understanding after looking through `SimpleVersionedSerializer` docs:
It seems like `SimpleVersionedSerializer` can only handle one version at a time (hence the `getVersion()` function).
Is that correct? Are there any best practices when working with `SimpleVersionedSerializer` to consider? When we evolve, will we have two classes or put all known classes into one instance?
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java
##########
@@ -34,7 +34,7 @@
/**
* Context object with optional arguments for a Flink Scan.
*/
-class ScanContext implements Serializable {
+public class ScanContext implements Serializable {
Review comment:
I believe the usage in `org.apache.iceberg.flink.source.split.SplitHelpers` is why this is now public instead of package-private.
Is it abnormal for various `Context` objects in Flink to be public?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r766918394
##########
File path: flink/v1.13/flink/src/test/java/org/apache/iceberg/flink/source/split/SplitHelpers.java
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import org.apache.iceberg.BaseCombinedScanTask;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MockFileScanTask;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericAppenderHelper;
+import org.apache.iceberg.data.RandomGenericData;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.flink.TestFixtures;
+import org.apache.iceberg.flink.source.FlinkSplitPlanner;
+import org.apache.iceberg.flink.source.ScanContext;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.Assert;
+import org.junit.rules.TemporaryFolder;
+
+public class SplitHelpers {
+
+ private static final AtomicLong splitLengthIncrement = new AtomicLong();
+
+ private SplitHelpers() {
+
+ }
+
+ public static List<IcebergSourceSplit> createMockedSplits(int splitCount) {
+ List<IcebergSourceSplit> splits = new ArrayList<>();
+ for (int i = 0; i < splitCount; ++i) {
+ // make sure each task has a different length,
+ // as it is part of the splitId calculation.
+ // This way, we can make sure all generated splits have different splitIds
+ FileScanTask fileScanTask = new MockFileScanTask(1024 + splitLengthIncrement.incrementAndGet());
+ CombinedScanTask combinedScanTask = new BaseCombinedScanTask(fileScanTask);
+ splits.add(IcebergSourceSplit.fromCombinedScanTask(combinedScanTask));
+ }
+ return splits;
+ }
+
+ /**
+ * Unlike {@link SplitHelpers#createMockedSplits(int)} above, this is to generate some
+ * realistic IcebergSourceSplit with actual file paths.
+ *
+ * Actual data files are already deleted before return. Caller shouldn't attempt to read the
+ * data files. They are intended for serializer or enumerator unit test (without actual reading).
+ */
+ public static List<IcebergSourceSplit> createFileSplits(
+ TemporaryFolder temporaryFolder, int fileCount, int filesPerSplit) throws Exception {
+ File warehouseFile = temporaryFolder.newFolder();
+ Assert.assertTrue(warehouseFile.delete());
+ String warehouse = "file:" + warehouseFile;
+ org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
Review comment:
Can we remove the fully-qualified path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r766919061
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/Position.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+/**
+ * A mutable class that defines the read position
+ * <ul>
+ * <li>file offset in the list of files in a {@link CombinedScanTask}</li>
+ * <li>record offset within a file</li>
+ * </ul>
+ */
+@Internal
+public class Position implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ private int fileOffset;
+ private long recordOffset;
+
+ public Position(int fileOffset, long recordOffset) {
+ this.fileOffset = fileOffset;
+ this.recordOffset = recordOffset;
+ }
+
+ void advanceFile() {
+ this.fileOffset += 1;
+ this.recordOffset = 0L;
+ }
+
+ void advanceRecord() {
+ this.recordOffset += 1L;
+ }
+
+ public void update(int newFileOffset, long newRecordOffset) {
+ this.fileOffset = newFileOffset;
+ this.recordOffset = newRecordOffset;
+ }
+
+ public int fileOffset() {
+ return fileOffset;
+ }
+
+ public long recordOffset() {
+ return recordOffset;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ Position that = (Position) o;
+ return Objects.equals(fileOffset, that.fileOffset) &&
+ Objects.equals(recordOffset, that.recordOffset);
Review comment:
I tend to prefer `Object.equals` as it immediately does `==` check anyway, and then I don't have to think about it in the future.
If the project style prefers `==` where possible, then by all means go with that.
Object.equals code per OpenJDK 8.
```java
public static boolean equals(Object a, Object b) {
return (a == b) || (a != null && a.equals(b));
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r766903328
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java
##########
@@ -34,7 +34,7 @@
/**
* Context object with optional arguments for a Flink Scan.
*/
-class ScanContext implements Serializable {
+public class ScanContext implements Serializable {
Review comment:
I think we should just have the minimal set of changes to this file, rather than trying to fix the duplication between this and `FlinkSource.Builder`. Also, we should probably add `@Internal` because this is becoming public.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r766898490
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/Position.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+/**
+ * A mutable class that defines the read position
+ * <ul>
+ * <li>file offset in the list of files in a {@link CombinedScanTask}</li>
+ * <li>record offset within a file</li>
+ * </ul>
+ */
+@Internal
+public class Position implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ private int fileOffset;
+ private long recordOffset;
+
+ public Position(int fileOffset, long recordOffset) {
+ this.fileOffset = fileOffset;
+ this.recordOffset = recordOffset;
+ }
+
+ void advanceFile() {
+ this.fileOffset += 1;
+ this.recordOffset = 0L;
+ }
+
+ void advanceRecord() {
+ this.recordOffset += 1L;
+ }
+
+ public void update(int newFileOffset, long newRecordOffset) {
+ this.fileOffset = newFileOffset;
+ this.recordOffset = newRecordOffset;
+ }
+
+ public int fileOffset() {
+ return fileOffset;
+ }
+
+ public long recordOffset() {
+ return recordOffset;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ Position that = (Position) o;
+ return Objects.equals(fileOffset, that.fileOffset) &&
+ Objects.equals(recordOffset, that.recordOffset);
Review comment:
These are both primitives, so you can use `==` instead of `Objects.equals`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r766895129
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java
##########
@@ -34,7 +34,7 @@
/**
* Context object with optional arguments for a Flink Scan.
*/
-class ScanContext implements Serializable {
+public class ScanContext implements Serializable {
Review comment:
I'm good with not having to duplicate setters in `FlinkSource` đź‘Ť
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r750200159
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final CombinedScanTask task;
+ /**
Review comment:
Nit: Could you pls leave a separate empty line between the two different block ?
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/FlinkSplitPlanner.java
##########
@@ -22,33 +22,58 @@
import java.io.IOException;
import java.io.UncheckedIOException;
import java.util.List;
+import org.apache.flink.annotation.Internal;
import org.apache.iceberg.CombinedScanTask;
import org.apache.iceberg.Table;
import org.apache.iceberg.TableProperties;
import org.apache.iceberg.TableScan;
import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.flink.source.split.IcebergSourceSplit;
import org.apache.iceberg.io.CloseableIterable;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-class FlinkSplitGenerator {
- private FlinkSplitGenerator() {
+@Internal
+public class FlinkSplitPlanner {
+ private FlinkSplitPlanner() {
}
- static FlinkInputSplit[] createInputSplits(Table table, ScanContext context) {
- List<CombinedScanTask> tasks = tasks(table, context);
- FlinkInputSplit[] splits = new FlinkInputSplit[tasks.size()];
- for (int i = 0; i < tasks.size(); i++) {
- splits[i] = new FlinkInputSplit(i, tasks.get(i));
+ static FlinkInputSplit[] planInputSplits(Table table, ScanContext context) {
+ try (CloseableIterable<CombinedScanTask> tasksIterable = planTasks(table, context)) {
+ List<CombinedScanTask> tasks = Lists.newArrayList(tasksIterable);
+ FlinkInputSplit[] splits = new FlinkInputSplit[tasks.size()];
+ for (int i = 0; i < tasks.size(); i++) {
+ splits[i] = new FlinkInputSplit(i, tasks.get(i));
+ }
+ return splits;
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to process tasks iterable", e);
+ }
+ }
+
+ /**
+ * This returns splits for the FLIP-27 source
+ */
+ public static List<IcebergSourceSplit> planIcebergSourceSplits(
+ Table table, ScanContext context) {
+ try (CloseableIterable<CombinedScanTask> tasksIterable = planTasks(table, context)) {
+ List<IcebergSourceSplit> splits = Lists.newArrayList();
+ tasksIterable.forEach(task -> splits.add(IcebergSourceSplit.fromCombinedScanTask(task)));
+ return splits;
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to process task iterable: ", e);
}
- return splits;
}
- private static List<CombinedScanTask> tasks(Table table, ScanContext context) {
+ static CloseableIterable<CombinedScanTask> planTasks(Table table, ScanContext context) {
TableScan scan = table
.newScan()
.caseSensitive(context.caseSensitive())
.project(context.project());
+ if (context.includeColumnStats()) {
+ scan = scan.includeColumnStats();
+ }
Review comment:
Why add this switch in this PR ?
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplitSerializer.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.IOException;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.util.InstantiationUtil;
+
+/**
+ * TODO: use Java serialization for now.
+ * Will switch to more stable serializer from
+ * <a href="https://github.com/apache/iceberg/issues/1698">issue-1698</a>.
+ */
+@Internal
+public class IcebergSourceSplitSerializer implements SimpleVersionedSerializer<IcebergSourceSplit> {
+ public static final IcebergSourceSplitSerializer INSTANCE = new IcebergSourceSplitSerializer();
+ private static final int VERSION = 1;
+
+ @Override
+ public int getVersion() {
+ return VERSION;
+ }
+
+ @Override
+ public byte[] serialize(IcebergSourceSplit split) throws IOException {
+ if (split.serializedBytesCache() == null) {
+ final byte[] result = serializeV1(split);
Review comment:
Nit: remove the unnecessary `final` modifier.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/FlinkSplitPlanner.java
##########
@@ -22,33 +22,58 @@
import java.io.IOException;
import java.io.UncheckedIOException;
import java.util.List;
+import org.apache.flink.annotation.Internal;
import org.apache.iceberg.CombinedScanTask;
import org.apache.iceberg.Table;
import org.apache.iceberg.TableProperties;
import org.apache.iceberg.TableScan;
import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.flink.source.split.IcebergSourceSplit;
import org.apache.iceberg.io.CloseableIterable;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-class FlinkSplitGenerator {
- private FlinkSplitGenerator() {
+@Internal
+public class FlinkSplitPlanner {
+ private FlinkSplitPlanner() {
}
- static FlinkInputSplit[] createInputSplits(Table table, ScanContext context) {
- List<CombinedScanTask> tasks = tasks(table, context);
- FlinkInputSplit[] splits = new FlinkInputSplit[tasks.size()];
- for (int i = 0; i < tasks.size(); i++) {
- splits[i] = new FlinkInputSplit(i, tasks.get(i));
+ static FlinkInputSplit[] planInputSplits(Table table, ScanContext context) {
+ try (CloseableIterable<CombinedScanTask> tasksIterable = planTasks(table, context)) {
+ List<CombinedScanTask> tasks = Lists.newArrayList(tasksIterable);
+ FlinkInputSplit[] splits = new FlinkInputSplit[tasks.size()];
+ for (int i = 0; i < tasks.size(); i++) {
+ splits[i] = new FlinkInputSplit(i, tasks.get(i));
+ }
+ return splits;
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to process tasks iterable", e);
+ }
+ }
+
+ /**
+ * This returns splits for the FLIP-27 source
+ */
+ public static List<IcebergSourceSplit> planIcebergSourceSplits(
+ Table table, ScanContext context) {
+ try (CloseableIterable<CombinedScanTask> tasksIterable = planTasks(table, context)) {
+ List<IcebergSourceSplit> splits = Lists.newArrayList();
+ tasksIterable.forEach(task -> splits.add(IcebergSourceSplit.fromCombinedScanTask(task)));
+ return splits;
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to process task iterable: ", e);
}
- return splits;
}
- private static List<CombinedScanTask> tasks(Table table, ScanContext context) {
+ static CloseableIterable<CombinedScanTask> planTasks(Table table, ScanContext context) {
TableScan scan = table
.newScan()
.caseSensitive(context.caseSensitive())
.project(context.project());
+ if (context.includeColumnStats()) {
+ scan = scan.includeColumnStats();
+ }
Review comment:
I will recommend to make this PR to add the flip-27 source split as focused as possible. So it will recommend to remove the unrelated changes.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/FlinkSplitPlanner.java
##########
@@ -22,33 +22,58 @@
import java.io.IOException;
import java.io.UncheckedIOException;
import java.util.List;
+import org.apache.flink.annotation.Internal;
import org.apache.iceberg.CombinedScanTask;
import org.apache.iceberg.Table;
import org.apache.iceberg.TableProperties;
import org.apache.iceberg.TableScan;
import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.flink.source.split.IcebergSourceSplit;
import org.apache.iceberg.io.CloseableIterable;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-class FlinkSplitGenerator {
- private FlinkSplitGenerator() {
+@Internal
+public class FlinkSplitPlanner {
+ private FlinkSplitPlanner() {
}
- static FlinkInputSplit[] createInputSplits(Table table, ScanContext context) {
- List<CombinedScanTask> tasks = tasks(table, context);
- FlinkInputSplit[] splits = new FlinkInputSplit[tasks.size()];
- for (int i = 0; i < tasks.size(); i++) {
- splits[i] = new FlinkInputSplit(i, tasks.get(i));
+ static FlinkInputSplit[] planInputSplits(Table table, ScanContext context) {
+ try (CloseableIterable<CombinedScanTask> tasksIterable = planTasks(table, context)) {
+ List<CombinedScanTask> tasks = Lists.newArrayList(tasksIterable);
+ FlinkInputSplit[] splits = new FlinkInputSplit[tasks.size()];
+ for (int i = 0; i < tasks.size(); i++) {
+ splits[i] = new FlinkInputSplit(i, tasks.get(i));
+ }
+ return splits;
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to process tasks iterable", e);
+ }
+ }
+
+ /**
+ * This returns splits for the FLIP-27 source
+ */
+ public static List<IcebergSourceSplit> planIcebergSourceSplits(
Review comment:
Nit: I think we don't need to switch to a new line.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedFormCache;
+
+ public IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedFormCache() {
+ return serializedFormCache;
+ }
+
+ void serializedFormCache(byte[] cachedBytes) {
+ this.serializedFormCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ IcebergSourceSplit split = (IcebergSourceSplit) o;
+ return Objects.equal(splitId(), split.splitId()) &&
+ Objects.equal(position, split.position());
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hashCode(splitId());
Review comment:
Then why not assign an unique integer number to the `IcebergSourceSplit` as `splitId` when planing the tasks in `FlinkSplitPlanner#planIcebergSourceSplits` ? I think keeping the `toString` approach as the identifier did not answer the question I raised in the first comment.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedBytesCache;
+
+ private IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedBytesCache() {
+ return serializedBytesCache;
+ }
+
+ void serializedBytesCache(byte[] cachedBytes) {
+ this.serializedBytesCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
Review comment:
No usage for this method ?
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/Position.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+/**
+ * A mutable class that defines the read position
+ * <ul>
+ * <li>file offset in the list of files in a {@link CombinedScanTask}</li>
+ * <li>record offset within a file</li>
+ * </ul>
+ */
+@Internal
+public class Position implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ private int fileOffset;
+ private long recordOffset;
+
+ public Position(int fileOffset, long recordOffset) {
+ this.fileOffset = fileOffset;
+ this.recordOffset = recordOffset;
+ }
+
+ void advanceFile() {
+ this.fileOffset += 1;
+ this.recordOffset = 0L;
+ }
+
+ void advanceRecord() {
+ this.recordOffset += 1L;
+ }
+
+ public void update(int newFileOffset, long newRecordOffset) {
+ this.fileOffset = newFileOffset;
+ this.recordOffset = newRecordOffset;
+ }
+
+ public int fileOffset() {
+ return fileOffset;
+ }
+
+ public long recordOffset() {
+ return recordOffset;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ final Position that = (Position) o;
Review comment:
Nit: we usually don't use `final` for a local variable in iceberg.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java
##########
@@ -34,7 +34,7 @@
/**
* Context object with optional arguments for a Flink Scan.
*/
-class ScanContext implements Serializable {
+public class ScanContext implements Serializable {
Review comment:
It's okay to make a class from flink module to be public, but we need to choose them carefully because there are many flink users who are writing Java datastream job. Making a class to be public may imply that we need to consider the class compatibility for downstream users. So in theory I will recommend to not expose a class as `public` by default unless we have to.
To avoid this ScanContext to be public, we even introduced many fields setters which are the same as the `ScanContext#Builder` in `FlinkSource`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r750211751
##########
File path: flink/v1.13/flink/src/test/java/org/apache/iceberg/flink/source/split/SplitHelpers.java
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import org.apache.iceberg.BaseCombinedScanTask;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MockFileScanTask;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericAppenderHelper;
+import org.apache.iceberg.data.RandomGenericData;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.flink.TestFixtures;
+import org.apache.iceberg.flink.source.FlinkSplitPlanner;
+import org.apache.iceberg.flink.source.ScanContext;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.Assert;
+import org.junit.rules.TemporaryFolder;
+
+public class SplitHelpers {
+
+ private static final AtomicLong splitLengthIncrement = new AtomicLong();
+
+ private SplitHelpers() {
+
+ }
+
+ public static List<IcebergSourceSplit> createMockedSplits(int splitCount) {
+ final List<IcebergSourceSplit> splits = new ArrayList<>();
+ for (int i = 0; i < splitCount; ++i) {
+ // make sure each task has a different length,
+ // as it is part of the splitId calculation.
+ // This way, we can make sure all generated splits have different splitIds
+ FileScanTask fileScanTask = new MockFileScanTask(1024 + splitLengthIncrement.incrementAndGet());
+ CombinedScanTask combinedScanTask = new BaseCombinedScanTask(fileScanTask);
+ splits.add(IcebergSourceSplit.fromCombinedScanTask(combinedScanTask));
+ }
+ return splits;
+ }
+
+ public static List<IcebergSourceSplit> createFileSplits(
+ TemporaryFolder temporaryFolder, int fileCount, int filesPerSplit) throws Exception {
+ final File warehouseFile = temporaryFolder.newFolder();
+ Assert.assertTrue(warehouseFile.delete());
+ final String warehouse = "file:" + warehouseFile;
+ org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
+ final HadoopCatalog catalog = new HadoopCatalog(hadoopConf, warehouse);
+ try {
+ final Table table = catalog.createTable(TestFixtures.TABLE_IDENTIFIER, TestFixtures.SCHEMA);
+ final GenericAppenderHelper dataAppender = new GenericAppenderHelper(
+ table, FileFormat.PARQUET, temporaryFolder);
+ for (int i = 0; i < fileCount; ++i) {
+ List<Record> records = RandomGenericData.generate(TestFixtures.SCHEMA, 2, i);
+ dataAppender.appendToTable(records);
+ }
+
+ final ScanContext scanContext = ScanContext.builder().build();
+ final List<IcebergSourceSplit> splits = FlinkSplitPlanner.planIcebergSourceSplits(table, scanContext);
+ return splits.stream()
+ .flatMap(split -> {
+ List<List<FileScanTask>> filesList = Lists.partition(new ArrayList<>(split.task().files()), filesPerSplit);
+ return filesList.stream()
+ .map(files -> new BaseCombinedScanTask(files))
+ .map(combinedScanTask -> IcebergSourceSplit.fromCombinedScanTask(combinedScanTask));
Review comment:
It's good to use method lambda here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r745470696
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java
##########
@@ -34,7 +34,7 @@
/**
* Context object with optional arguments for a Flink Scan.
*/
-class ScanContext implements Serializable {
+public class ScanContext implements Serializable {
Review comment:
Is this change related to this PR ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ChethanUK commented on pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
ChethanUK commented on pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#issuecomment-991892836
@stevenzwu does this work with Flink 1.14?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#issuecomment-992081066
> @stevenzwu does this work with Flink 1.14?
This should work for 1.14. The convention in the repo is to do large PRs against one version and then back port in another PR (usually we apply it on the latest supported version, but this has existed for a while so latest at the time was 1.13).
Unless you’re aware of a specific reason this would not work wirh 1.14, I’m fairly certain it will given that we’re using the newer interfaces. And on top of that, we’re using one of the higher level split interfaces.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r766895353
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/FlinkSplitPlanner.java
##########
@@ -22,33 +22,57 @@
import java.io.IOException;
import java.io.UncheckedIOException;
import java.util.List;
+import org.apache.flink.annotation.Internal;
import org.apache.iceberg.CombinedScanTask;
import org.apache.iceberg.Table;
import org.apache.iceberg.TableProperties;
import org.apache.iceberg.TableScan;
import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.flink.source.split.IcebergSourceSplit;
import org.apache.iceberg.io.CloseableIterable;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-class FlinkSplitGenerator {
- private FlinkSplitGenerator() {
+@Internal
+public class FlinkSplitPlanner {
+ private FlinkSplitPlanner() {
}
- static FlinkInputSplit[] createInputSplits(Table table, ScanContext context) {
- List<CombinedScanTask> tasks = tasks(table, context);
- FlinkInputSplit[] splits = new FlinkInputSplit[tasks.size()];
- for (int i = 0; i < tasks.size(); i++) {
- splits[i] = new FlinkInputSplit(i, tasks.get(i));
+ static FlinkInputSplit[] planInputSplits(Table table, ScanContext context) {
+ try (CloseableIterable<CombinedScanTask> tasksIterable = planTasks(table, context)) {
+ List<CombinedScanTask> tasks = Lists.newArrayList(tasksIterable);
+ FlinkInputSplit[] splits = new FlinkInputSplit[tasks.size()];
+ for (int i = 0; i < tasks.size(); i++) {
+ splits[i] = new FlinkInputSplit(i, tasks.get(i));
+ }
+ return splits;
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to process tasks iterable", e);
+ }
+ }
+
+ /**
+ * This returns splits for the FLIP-27 source
+ */
+ public static List<IcebergSourceSplit> planIcebergSourceSplits(Table table, ScanContext context) {
+ try (CloseableIterable<CombinedScanTask> tasksIterable = planTasks(table, context)) {
+ List<IcebergSourceSplit> splits = Lists.newArrayList();
+ tasksIterable.forEach(task -> splits.add(IcebergSourceSplit.fromCombinedScanTask(task)));
+ return splits;
Review comment:
`Lists.transform` is another option here instead of separate lines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r771532224
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java
##########
@@ -34,7 +34,7 @@
/**
* Context object with optional arguments for a Flink Scan.
*/
-class ScanContext implements Serializable {
+public class ScanContext implements Serializable {
Review comment:
To clarify, the FLIP-27 source change wasn't trying to fix the duplication btw `ScanContext` and current `FlinkSource.Builder`. Existing code stays unchanged. I was just trying to avoid the duplication with the new FLIP-27 `IcebergSource.Builder` by making `ScanContext` user-facing public and letting users directly construct the `ScanContext`.
With that said, it seems that both @openinx and @rdblue prefer the current `FlinkSource.Builder` model where `ScanContext` APIs were duplicated in the source builder. I will make the change to follow the current pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r771892927
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedBytesCache;
+
+ private IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedBytesCache() {
+ return serializedBytesCache;
+ }
+
+ void serializedBytesCache(byte[] cachedBytes) {
+ this.serializedBytesCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ IcebergSourceSplit split = (IcebergSourceSplit) o;
+ return Objects.equal(splitId(), split.splitId()) &&
+ Objects.equal(position, split.position());
Review comment:
agree with the inconsistent problem. let me look into this. We do have places where those two overrides are used.
e.g.
```
public class IcebergEnumeratorState implements Serializable {
private final Map<IcebergSourceSplit, IcebergSourceSplitStatus> pendingSplits;
}
Also a few unit tests rely on the `equals` to verify deserialized objects are the same as the original one. I am looking to replace those with a `Comparator` impl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r745771768
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/Position.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
Review comment:
@openinx thank you for catching this and following up with a CI fix.
I was using another cmd for local check. I guess I should run `build` in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r747800314
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedBytesCache;
+
+ private IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedBytesCache() {
+ return serializedBytesCache;
+ }
+
+ void serializedBytesCache(byte[] cachedBytes) {
+ this.serializedBytesCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ IcebergSourceSplit split = (IcebergSourceSplit) o;
+ return Objects.equal(splitId(), split.splitId()) &&
+ Objects.equal(position, split.position());
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hashCode(splitId());
+ }
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .add("position", position)
+ .toString();
+ }
+
+ private String toString(Collection<FileScanTask> files) {
+ return Iterables.toString(files.stream().map(fileScanTask ->
+ MoreObjects.toStringHelper(fileScanTask)
+ .add("file", fileScanTask.file() != null ?
Review comment:
@openinx I have to add back the null check, because in some other tests (outside this sub PR) we are using `MockFileScanTask` with just length. In that case, file will be null.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplitStatus.java
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+public enum IcebergSourceSplitStatus {
Review comment:
This is not used by this PR. will remove
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java
##########
@@ -34,7 +34,7 @@
/**
* Context object with optional arguments for a Flink Scan.
*/
-class ScanContext implements Serializable {
+public class ScanContext implements Serializable {
Review comment:
We currently hide `ScanContext` by duplicating all 10+ setters in `FlinkSource`. I am not sure the value of hiding `ScanContext` in this case. Eventually, users need to deal with the same setters via `FlinkSource` indirectly. Hence in the new FLIP-27 source, I made `ScanContext` a user-facing public API.
If most people think it is better to hide `ScanContext` and duplicate the setters in `IcebergSource` (FLIP-27), I will make `IcebergSouce` the same style as the current `FlinkSource`. cc @rdblue @kbendick for additional inputs.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedFormCache;
+
+ public IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedFormCache() {
+ return serializedFormCache;
+ }
+
+ void serializedFormCache(byte[] cachedBytes) {
+ this.serializedFormCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ IcebergSourceSplit split = (IcebergSourceSplit) o;
+ return Objects.equal(splitId(), split.splitId()) &&
+ Objects.equal(position, split.position());
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hashCode(splitId());
Review comment:
I am not sure we want to assign an unique integer number as `splitId` (especially for long-running streaming jobs).
We need to checkpoint the splitId counter. what if we can't restore from checkpoint (e.g. due to corrupted checkpoint state)? It is probably better to compute the `splitId` based on the intrinsic properties of `IcebergSourceSplit` (like path, start, length of data files).
I thought your first comment is about depending on `FileScanTask#toString`, which I explained that it is not the case. We don't call `FileScanTask#toString`. Instead, we depend on `fileScanTask.file().path().toString()`. Maybe I misunderstood your first comment. Can you elaborate a little more on your concern?
##########
File path: flink/v1.13/flink/src/test/java/org/apache/iceberg/flink/source/split/SplitHelpers.java
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import org.apache.iceberg.BaseCombinedScanTask;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MockFileScanTask;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericAppenderHelper;
+import org.apache.iceberg.data.RandomGenericData;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.flink.TestFixtures;
+import org.apache.iceberg.flink.source.FlinkSplitPlanner;
+import org.apache.iceberg.flink.source.ScanContext;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.Assert;
+import org.junit.rules.TemporaryFolder;
+
+public class SplitHelpers {
+
+ private static final AtomicLong splitLengthIncrement = new AtomicLong();
+
+ private SplitHelpers() {
+
+ }
+
+ public static List<IcebergSourceSplit> createMockedSplits(int splitCount) {
+ final List<IcebergSourceSplit> splits = new ArrayList<>();
+ for (int i = 0; i < splitCount; ++i) {
+ // make sure each task has a different length,
+ // as it is part of the splitId calculation.
+ // This way, we can make sure all generated splits have different splitIds
+ FileScanTask fileScanTask = new MockFileScanTask(1024 + splitLengthIncrement.incrementAndGet());
+ CombinedScanTask combinedScanTask = new BaseCombinedScanTask(fileScanTask);
+ splits.add(IcebergSourceSplit.fromCombinedScanTask(combinedScanTask));
+ }
+ return splits;
+ }
+
+ public static List<IcebergSourceSplit> createFileSplits(
+ TemporaryFolder temporaryFolder, int fileCount, int filesPerSplit) throws Exception {
+ final File warehouseFile = temporaryFolder.newFolder();
+ Assert.assertTrue(warehouseFile.delete());
+ final String warehouse = "file:" + warehouseFile;
+ org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
+ final HadoopCatalog catalog = new HadoopCatalog(hadoopConf, warehouse);
+ try {
+ final Table table = catalog.createTable(TestFixtures.TABLE_IDENTIFIER, TestFixtures.SCHEMA);
+ final GenericAppenderHelper dataAppender = new GenericAppenderHelper(
+ table, FileFormat.PARQUET, temporaryFolder);
+ for (int i = 0; i < fileCount; ++i) {
+ List<Record> records = RandomGenericData.generate(TestFixtures.SCHEMA, 2, i);
+ dataAppender.appendToTable(records);
+ }
+
+ final ScanContext scanContext = ScanContext.builder().build();
+ final List<IcebergSourceSplit> splits = FlinkSplitPlanner.planIcebergSourceSplits(table, scanContext);
+ return splits.stream()
+ .flatMap(split -> {
+ List<List<FileScanTask>> filesList = Lists.partition(new ArrayList<>(split.task().files()), filesPerSplit);
+ return filesList.stream()
+ .map(files -> new BaseCombinedScanTask(files))
+ .map(combinedScanTask -> IcebergSourceSplit.fromCombinedScanTask(combinedScanTask));
+ })
+ .collect(Collectors.toList());
+ } finally {
+ catalog.dropTable(TestFixtures.TABLE_IDENTIFIER);
Review comment:
Good question. I see why this can be confusing.
Right now, this method is used only by `TestIcebergSourceSplitSerializer` to generate some realistic splits with actual paths (unlike the `createMockedSplits`). I will add some comments both at method level and here. Let me know if I should move this method into the `TestIcebergSourceSplitSerializer`. I kept it here so that it might be useful for other testing
##########
File path: flink/v1.13/flink/src/test/java/org/apache/iceberg/flink/source/split/SplitHelpers.java
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import org.apache.iceberg.BaseCombinedScanTask;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MockFileScanTask;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericAppenderHelper;
+import org.apache.iceberg.data.RandomGenericData;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.flink.TestFixtures;
+import org.apache.iceberg.flink.source.FlinkSplitPlanner;
+import org.apache.iceberg.flink.source.ScanContext;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.Assert;
+import org.junit.rules.TemporaryFolder;
+
+public class SplitHelpers {
+
+ private static final AtomicLong splitLengthIncrement = new AtomicLong();
+
+ private SplitHelpers() {
+
+ }
+
+ public static List<IcebergSourceSplit> createMockedSplits(int splitCount) {
+ final List<IcebergSourceSplit> splits = new ArrayList<>();
+ for (int i = 0; i < splitCount; ++i) {
+ // make sure each task has a different length,
+ // as it is part of the splitId calculation.
+ // This way, we can make sure all generated splits have different splitIds
+ FileScanTask fileScanTask = new MockFileScanTask(1024 + splitLengthIncrement.incrementAndGet());
+ CombinedScanTask combinedScanTask = new BaseCombinedScanTask(fileScanTask);
+ splits.add(IcebergSourceSplit.fromCombinedScanTask(combinedScanTask));
+ }
+ return splits;
+ }
+
+ public static List<IcebergSourceSplit> createFileSplits(
+ TemporaryFolder temporaryFolder, int fileCount, int filesPerSplit) throws Exception {
+ final File warehouseFile = temporaryFolder.newFolder();
+ Assert.assertTrue(warehouseFile.delete());
+ final String warehouse = "file:" + warehouseFile;
+ org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
+ final HadoopCatalog catalog = new HadoopCatalog(hadoopConf, warehouse);
+ try {
+ final Table table = catalog.createTable(TestFixtures.TABLE_IDENTIFIER, TestFixtures.SCHEMA);
+ final GenericAppenderHelper dataAppender = new GenericAppenderHelper(
+ table, FileFormat.PARQUET, temporaryFolder);
+ for (int i = 0; i < fileCount; ++i) {
+ List<Record> records = RandomGenericData.generate(TestFixtures.SCHEMA, 2, i);
+ dataAppender.appendToTable(records);
+ }
+
+ final ScanContext scanContext = ScanContext.builder().build();
+ final List<IcebergSourceSplit> splits = FlinkSplitPlanner.planIcebergSourceSplits(table, scanContext);
+ return splits.stream()
+ .flatMap(split -> {
+ List<List<FileScanTask>> filesList = Lists.partition(new ArrayList<>(split.task().files()), filesPerSplit);
+ return filesList.stream()
+ .map(files -> new BaseCombinedScanTask(files))
+ .map(combinedScanTask -> IcebergSourceSplit.fromCombinedScanTask(combinedScanTask));
Review comment:
will do
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java
##########
@@ -34,7 +34,7 @@
/**
* Context object with optional arguments for a Flink Scan.
*/
-class ScanContext implements Serializable {
+public class ScanContext implements Serializable {
Review comment:
Once SplitEnumerator classes (not included in this PR) will also use ScanContext and FlinkSplitPlanner. enumerator will be in a diff sub package. Hence FlinkSplitPlanner will need to be public too. But FlinkSplitPlanner is marked as @Internal.
Current [FlinkSource](https://github.com/stevenzwu/iceberg/blob/flip27IcebergSource/flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/FlinkSource.java) clone all `ScanContext` APIs in its builder, which I think is unnecessary.
`ScanContext` will be a public API for the FLIP-27 [IcebergSource](https://github.com/stevenzwu/iceberg/blob/flip27IcebergSource/flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/IcebergSource.java#L181)
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
Review comment:
good catch. will add it
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedFormCache;
+
+ public IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedFormCache() {
+ return serializedFormCache;
+ }
+
+ void serializedFormCache(byte[] cachedBytes) {
+ this.serializedFormCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ IcebergSourceSplit split = (IcebergSourceSplit) o;
+ return Objects.equal(splitId(), split.splitId()) &&
+ Objects.equal(position, split.position());
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hashCode(splitId());
Review comment:
@kbendick you are correct. I had a typo in my earlier comment. should be "can't use the lazy approach because ..."
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedFormCache;
+
+ public IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedFormCache() {
+ return serializedFormCache;
+ }
+
+ void serializedFormCache(byte[] cachedBytes) {
+ this.serializedFormCache = cachedBytes;
+ }
Review comment:
Serialization is done by another class `IcebergSourceSplitSerializer`. It can't be computed and cached internally. Hence this setter provides a way for `IcebergSourceSplitSerializer` to cache the serialized bytes.
`lazyIdToField ` can work because everything is encapsulated within the `Schema` class.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedFormCache;
+
+ public IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedFormCache() {
+ return serializedFormCache;
+ }
+
+ void serializedFormCache(byte[] cachedBytes) {
+ this.serializedFormCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ IcebergSourceSplit split = (IcebergSourceSplit) o;
+ return Objects.equal(splitId(), split.splitId()) &&
+ Objects.equal(position, split.position());
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hashCode(splitId());
+ }
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .add("position", position)
+ .toString();
+ }
+
+ private String toString(Collection<FileScanTask> files) {
+ return Iterables.toString(files.stream().map(fileScanTask ->
+ MoreObjects.toStringHelper(fileScanTask)
+ .add("file", fileScanTask.file() != null ?
+ fileScanTask.file().path().toString() :
+ "NoFile")
Review comment:
will remove the null check here
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedFormCache;
+
+ public IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedFormCache() {
+ return serializedFormCache;
+ }
+
+ void serializedFormCache(byte[] cachedBytes) {
+ this.serializedFormCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ IcebergSourceSplit split = (IcebergSourceSplit) o;
+ return Objects.equal(splitId(), split.splitId()) &&
+ Objects.equal(position, split.position());
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hashCode(splitId());
Review comment:
as mentioned in the other comment, `serializedFormCache` can't use the lazy approach because the serialization is done by a separate serializer class. `IcebergSourceSplit` doesn't know how to serialize itself. Hence we can't calculate the byte array's hashCode.
We aren't not using the `toString` from `FileScanTask`. We are only using `fileScanTask.file().path().toString()`.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedFormCache;
+
+ public IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedFormCache() {
+ return serializedFormCache;
+ }
+
+ void serializedFormCache(byte[] cachedBytes) {
+ this.serializedFormCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ IcebergSourceSplit split = (IcebergSourceSplit) o;
+ return Objects.equal(splitId(), split.splitId()) &&
+ Objects.equal(position, split.position());
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hashCode(splitId());
Review comment:
@openinx not sure where is the miscommunication. but we don't call `FileScanTask#toString` here. Instead, we only call `FileScanTask.file().path().toString()`. Please see this code snippet below.
```
private String toString(Collection<FileScanTask> files) {
return Iterables.toString(files.stream().map(fileScanTask ->
MoreObjects.toStringHelper(fileScanTask)
.add("file", fileScanTask.file() != null ?
fileScanTask.file().path().toString() :
"NoDataFile")
.add("start", fileScanTask.start())
.add("length", fileScanTask.length())
.toString()).collect(Collectors.toList()));
}
```
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedFormCache;
+
+ public IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedFormCache() {
+ return serializedFormCache;
+ }
+
+ void serializedFormCache(byte[] cachedBytes) {
+ this.serializedFormCache = cachedBytes;
+ }
Review comment:
To me, lazy pattern is encapsulated within the class. E.g., `Schema` class knows how to compute the IdToField mapping in the `lazyIdToField` method. Here, we have a more getter/setter protocol btw two classes for caching serialized bytes.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplitSerializer.java
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.IOException;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.util.InstantiationUtil;
+
+/**
+ * TODO: use Java serialization for now.
+ * Will switch to more stable serializer from
+ * <a href="https://github.com/apache/iceberg/issues/1698">issue-1698</a>.
+ */
+@Internal
+public class IcebergSourceSplitSerializer implements SimpleVersionedSerializer<IcebergSourceSplit> {
Review comment:
`SimpleVersionedSerializer` will always use one/latest version to serialize. But during deserialization, it should handle multiple versions to support evolution (e.g. when we switch from Java serializable to some Avro serialization for FileScanTask)
```
@Override
public IcebergSourceSplit deserialize(int version, byte[] serialized) throws IOException {
switch (version) {
case 1:
return deserializeV1(serialized);
default:
throw new IOException("Unknown version: " + version);
}
}
```
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final CombinedScanTask task;
+ /**
Review comment:
There are two `private final` variables. Hence I thought it is one block. it is just that the 2nd variable has a Javadoc. If it is Iceberg's style convention to always start a Javadoc comment with an empty line, I am very happy to conform
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java
##########
@@ -34,7 +34,7 @@
/**
* Context object with optional arguments for a Flink Scan.
*/
-class ScanContext implements Serializable {
+public class ScanContext implements Serializable {
Review comment:
I basically included all classes (including test) connected to the `IcebergSourceSplit`. I then copied those classes from the uber `flip27IcebergSource` branch to this new branch.
`ScanContext` is used by `FlinkSplitPlanner`.
```
if (context.includeColumnStats()) {
scan = scan.includeColumnStats();
}
```
`FlinkSplitPlanner` (renamed from `FlinkSplitGenerator`) is used by `SplitHelper` test class
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplitSerializer.java
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.IOException;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.util.InstantiationUtil;
+
+/**
+ * TODO: use Java serialization for now.
+ * Will switch to more stable serializer from
+ * <a href="https://github.com/apache/iceberg/issues/1698">issue-1698</a>.
+ */
+@Internal
+public class IcebergSourceSplitSerializer implements SimpleVersionedSerializer<IcebergSourceSplit> {
+ public static final IcebergSourceSplitSerializer INSTANCE = new IcebergSourceSplitSerializer();
+ private static final int VERSION = 1;
+
+ @Override
+ public int getVersion() {
+ return VERSION;
+ }
+
+ @Override
+ public byte[] serialize(IcebergSourceSplit split) throws IOException {
+ if (split.serializedFormCache() == null) {
+ final byte[] result = serializeV1(split);
+ split.serializedFormCache(result);
+ }
+ return split.serializedFormCache();
+ }
+
+ @Override
+ public IcebergSourceSplit deserialize(int version, byte[] serialized) throws IOException {
+ switch (version) {
+ case 1:
+ return deserializeV1(serialized);
+ default:
+ throw new IOException("Unknown version: " + version);
Review comment:
"maximum version" is probably also not accurate. it implies all versions below this number are supported. We may drop support of deserializing older versions. I will change the error msg to sth like
```
Failed to deserialize IcebergSourceSplit. Encountered unsupported version: $version. Supported version are [1]"
```
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedFormCache;
Review comment:
will change it to `serializedBytesCache`. This is not a "lazy" pattern, which is typically encapsulated within the class. this is a getter/setter interaction btw `IcebergSourceSplit` and `IcebergSourceSplitSerializer`
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/Position.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+/**
+ * A mutable class that defines the read position
+ * <ul>
+ * <li>file offset in the list of files in a {@link CombinedScanTask}</li>
+ * <li>record offset within a file</li>
+ * </ul>
+ */
+@Internal
+public class Position implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ private int fileOffset;
+ private long recordOffset;
+
+ public Position(int fileOffset, long recordOffset) {
+ this.fileOffset = fileOffset;
+ this.recordOffset = recordOffset;
+ }
+
+ void advanceFile() {
+ this.fileOffset += 1;
+ this.recordOffset = 0L;
+ }
+
+ void advanceRecord() {
+ this.recordOffset += 1L;
+ }
+
+ public void update(int newFileOffset, long newRecordOffset) {
+ this.fileOffset = newFileOffset;
+ this.recordOffset = newRecordOffset;
+ }
+
+ public int fileOffset() {
+ return fileOffset;
+ }
+
+ public long recordOffset() {
+ return recordOffset;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ final Position that = (Position) o;
Review comment:
yes. will remove
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/FlinkSplitPlanner.java
##########
@@ -22,33 +22,58 @@
import java.io.IOException;
import java.io.UncheckedIOException;
import java.util.List;
+import org.apache.flink.annotation.Internal;
import org.apache.iceberg.CombinedScanTask;
import org.apache.iceberg.Table;
import org.apache.iceberg.TableProperties;
import org.apache.iceberg.TableScan;
import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.flink.source.split.IcebergSourceSplit;
import org.apache.iceberg.io.CloseableIterable;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-class FlinkSplitGenerator {
- private FlinkSplitGenerator() {
+@Internal
+public class FlinkSplitPlanner {
+ private FlinkSplitPlanner() {
}
- static FlinkInputSplit[] createInputSplits(Table table, ScanContext context) {
- List<CombinedScanTask> tasks = tasks(table, context);
- FlinkInputSplit[] splits = new FlinkInputSplit[tasks.size()];
- for (int i = 0; i < tasks.size(); i++) {
- splits[i] = new FlinkInputSplit(i, tasks.get(i));
+ static FlinkInputSplit[] planInputSplits(Table table, ScanContext context) {
+ try (CloseableIterable<CombinedScanTask> tasksIterable = planTasks(table, context)) {
+ List<CombinedScanTask> tasks = Lists.newArrayList(tasksIterable);
+ FlinkInputSplit[] splits = new FlinkInputSplit[tasks.size()];
+ for (int i = 0; i < tasks.size(); i++) {
+ splits[i] = new FlinkInputSplit(i, tasks.get(i));
+ }
+ return splits;
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to process tasks iterable", e);
+ }
+ }
+
+ /**
+ * This returns splits for the FLIP-27 source
+ */
+ public static List<IcebergSourceSplit> planIcebergSourceSplits(
+ Table table, ScanContext context) {
+ try (CloseableIterable<CombinedScanTask> tasksIterable = planTasks(table, context)) {
+ List<IcebergSourceSplit> splits = Lists.newArrayList();
+ tasksIterable.forEach(task -> splits.add(IcebergSourceSplit.fromCombinedScanTask(task)));
+ return splits;
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to process task iterable: ", e);
}
- return splits;
}
- private static List<CombinedScanTask> tasks(Table table, ScanContext context) {
+ static CloseableIterable<CombinedScanTask> planTasks(Table table, ScanContext context) {
TableScan scan = table
.newScan()
.caseSensitive(context.caseSensitive())
.project(context.project());
+ if (context.includeColumnStats()) {
+ scan = scan.includeColumnStats();
+ }
Review comment:
column stats are needed for event time/watermark aligned assigner. You are correct that it is directly used by this PR. Right now, I am taking the approach of splitting sub PRs at minimally connected files for easier creation of the sub PRs. if you think it is important to avoid unrelated changes inside a file, I can revert the piece of change.
##########
File path: flink/v1.13/flink/src/test/java/org/apache/iceberg/flink/source/split/SplitHelpers.java
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import org.apache.iceberg.BaseCombinedScanTask;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MockFileScanTask;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericAppenderHelper;
+import org.apache.iceberg.data.RandomGenericData;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.flink.TestFixtures;
+import org.apache.iceberg.flink.source.FlinkSplitPlanner;
+import org.apache.iceberg.flink.source.ScanContext;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.Assert;
+import org.junit.rules.TemporaryFolder;
+
+public class SplitHelpers {
+
+ private static final AtomicLong splitLengthIncrement = new AtomicLong();
+
+ private SplitHelpers() {
+
+ }
+
+ public static List<IcebergSourceSplit> createMockedSplits(int splitCount) {
+ final List<IcebergSourceSplit> splits = new ArrayList<>();
+ for (int i = 0; i < splitCount; ++i) {
+ // make sure each task has a different length,
+ // as it is part of the splitId calculation.
+ // This way, we can make sure all generated splits have different splitIds
+ FileScanTask fileScanTask = new MockFileScanTask(1024 + splitLengthIncrement.incrementAndGet());
+ CombinedScanTask combinedScanTask = new BaseCombinedScanTask(fileScanTask);
+ splits.add(IcebergSourceSplit.fromCombinedScanTask(combinedScanTask));
+ }
+ return splits;
+ }
+
+ public static List<IcebergSourceSplit> createFileSplits(
+ TemporaryFolder temporaryFolder, int fileCount, int filesPerSplit) throws Exception {
+ final File warehouseFile = temporaryFolder.newFolder();
Review comment:
thx. will do a complete pass on finding and removing the unnecessary `final` modifiers
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplitSerializer.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.IOException;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.util.InstantiationUtil;
+
+/**
+ * TODO: use Java serialization for now.
+ * Will switch to more stable serializer from
+ * <a href="https://github.com/apache/iceberg/issues/1698">issue-1698</a>.
+ */
+@Internal
+public class IcebergSourceSplitSerializer implements SimpleVersionedSerializer<IcebergSourceSplit> {
+ public static final IcebergSourceSplitSerializer INSTANCE = new IcebergSourceSplitSerializer();
+ private static final int VERSION = 1;
+
+ @Override
+ public int getVersion() {
+ return VERSION;
+ }
+
+ @Override
+ public byte[] serialize(IcebergSourceSplit split) throws IOException {
+ if (split.serializedBytesCache() == null) {
+ final byte[] result = serializeV1(split);
Review comment:
will fix
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/FlinkSplitPlanner.java
##########
@@ -22,33 +22,58 @@
import java.io.IOException;
import java.io.UncheckedIOException;
import java.util.List;
+import org.apache.flink.annotation.Internal;
import org.apache.iceberg.CombinedScanTask;
import org.apache.iceberg.Table;
import org.apache.iceberg.TableProperties;
import org.apache.iceberg.TableScan;
import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.flink.source.split.IcebergSourceSplit;
import org.apache.iceberg.io.CloseableIterable;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-class FlinkSplitGenerator {
- private FlinkSplitGenerator() {
+@Internal
+public class FlinkSplitPlanner {
+ private FlinkSplitPlanner() {
}
- static FlinkInputSplit[] createInputSplits(Table table, ScanContext context) {
- List<CombinedScanTask> tasks = tasks(table, context);
- FlinkInputSplit[] splits = new FlinkInputSplit[tasks.size()];
- for (int i = 0; i < tasks.size(); i++) {
- splits[i] = new FlinkInputSplit(i, tasks.get(i));
+ static FlinkInputSplit[] planInputSplits(Table table, ScanContext context) {
+ try (CloseableIterable<CombinedScanTask> tasksIterable = planTasks(table, context)) {
+ List<CombinedScanTask> tasks = Lists.newArrayList(tasksIterable);
+ FlinkInputSplit[] splits = new FlinkInputSplit[tasks.size()];
+ for (int i = 0; i < tasks.size(); i++) {
+ splits[i] = new FlinkInputSplit(i, tasks.get(i));
+ }
+ return splits;
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to process tasks iterable", e);
+ }
+ }
+
+ /**
+ * This returns splits for the FLIP-27 source
+ */
+ public static List<IcebergSourceSplit> planIcebergSourceSplits(
Review comment:
will fix
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedBytesCache;
+
+ private IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedBytesCache() {
+ return serializedBytesCache;
+ }
+
+ void serializedBytesCache(byte[] cachedBytes) {
+ this.serializedBytesCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
Review comment:
it is not used here. But it is used by `IcebergSourceRecordEmitter`.
Again, I am creating the PR for minimal connected files (not minimally connected code within files).
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplitSerializer.java
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.IOException;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.util.InstantiationUtil;
+
+/**
+ * TODO: use Java serialization for now.
+ * Will switch to more stable serializer from
+ * <a href="https://github.com/apache/iceberg/issues/1698">issue-1698</a>.
+ */
+@Internal
+public class IcebergSourceSplitSerializer implements SimpleVersionedSerializer<IcebergSourceSplit> {
Review comment:
This class is used for checkpoint state serializer. cross-process (JM->TM) communication is via Java serializable. Currently, we are using the Java serializable inside this class too for simpler start. This is not ideal, as we know Java serializable is not good with schema evolution. Schema evolution would be important for long-running streaming jobs (not so much for batch jobs).
In the class Javadoc, we linked to an issue for future improvement. Note that this is already an issue for the current `FlinkSource` in streaming mode.
https://github.com/apache/iceberg/issues/1698
##########
File path: flink/v1.13/flink/src/test/java/org/apache/iceberg/flink/source/split/TestIcebergSourceSplitSerializer.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.junit.Assert;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+public class TestIcebergSourceSplitSerializer {
+
+ @ClassRule
+ public static final TemporaryFolder TEMPORARY_FOLDER = new TemporaryFolder();
+
+ private final IcebergSourceSplitSerializer serializer = IcebergSourceSplitSerializer.INSTANCE;
+
+ @Test
+ public void testLatestVersion() throws Exception {
+ serializeAndDeserialize(1, 1);
+ serializeAndDeserialize(10, 2);
+ }
+
+ private void serializeAndDeserialize(int splitCount, int filesPerSplit) throws Exception {
+ final List<IcebergSourceSplit> splits = SplitHelpers.createFileSplits(TEMPORARY_FOLDER, splitCount, filesPerSplit);
+ for (IcebergSourceSplit split : splits) {
+ final byte[] result = serializer.serialize(split);
+ final IcebergSourceSplit deserialized = serializer.deserialize(serializer.getVersion(), result);
+ Assert.assertEquals(split, deserialized);
+
+ final byte[] cachedResult = serializer.serialize(split);
+ Assert.assertSame(result, cachedResult);
+ final IcebergSourceSplit deserialized2 = serializer.deserialize(serializer.getVersion(), cachedResult);
+ Assert.assertEquals(split, deserialized2);
Review comment:
we already asserted `deserialized ` in line 48. this is to make sure `deserialized2` from the 2nd call from `serializer.deserialize` (cached bytes) still gets the same split.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r750214588
##########
File path: flink/v1.13/flink/src/test/java/org/apache/iceberg/flink/source/split/TestIcebergSourceSplitSerializer.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.junit.Assert;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+public class TestIcebergSourceSplitSerializer {
+
+ @ClassRule
+ public static final TemporaryFolder TEMPORARY_FOLDER = new TemporaryFolder();
+
+ private final IcebergSourceSplitSerializer serializer = IcebergSourceSplitSerializer.INSTANCE;
+
+ @Test
+ public void testLatestVersion() throws Exception {
+ serializeAndDeserialize(1, 1);
+ serializeAndDeserialize(10, 2);
+ }
+
+ private void serializeAndDeserialize(int splitCount, int filesPerSplit) throws Exception {
+ final List<IcebergSourceSplit> splits = SplitHelpers.createFileSplits(TEMPORARY_FOLDER, splitCount, filesPerSplit);
+ for (IcebergSourceSplit split : splits) {
+ final byte[] result = serializer.serialize(split);
+ final IcebergSourceSplit deserialized = serializer.deserialize(serializer.getVersion(), result);
+ Assert.assertEquals(split, deserialized);
+
+ final byte[] cachedResult = serializer.serialize(split);
+ Assert.assertSame(result, cachedResult);
+ final IcebergSourceSplit deserialized2 = serializer.deserialize(serializer.getVersion(), cachedResult);
+ Assert.assertEquals(split, deserialized2);
Review comment:
Why not just use the `deserialized` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r771893035
##########
File path: flink/v1.13/flink/src/test/java/org/apache/iceberg/flink/source/split/SplitHelpers.java
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import org.apache.iceberg.BaseCombinedScanTask;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MockFileScanTask;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericAppenderHelper;
+import org.apache.iceberg.data.RandomGenericData;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.flink.TestFixtures;
+import org.apache.iceberg.flink.source.FlinkSplitPlanner;
+import org.apache.iceberg.flink.source.ScanContext;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.Assert;
+import org.junit.rules.TemporaryFolder;
+
+public class SplitHelpers {
+
+ private static final AtomicLong splitLengthIncrement = new AtomicLong();
+
+ private SplitHelpers() {
+
+ }
+
+ public static List<IcebergSourceSplit> createMockedSplits(int splitCount) {
+ List<IcebergSourceSplit> splits = new ArrayList<>();
+ for (int i = 0; i < splitCount; ++i) {
+ // make sure each task has a different length,
+ // as it is part of the splitId calculation.
+ // This way, we can make sure all generated splits have different splitIds
+ FileScanTask fileScanTask = new MockFileScanTask(1024 + splitLengthIncrement.incrementAndGet());
+ CombinedScanTask combinedScanTask = new BaseCombinedScanTask(fileScanTask);
+ splits.add(IcebergSourceSplit.fromCombinedScanTask(combinedScanTask));
+ }
+ return splits;
+ }
+
+ /**
+ * Unlike {@link SplitHelpers#createMockedSplits(int)} above, this is to generate some
+ * realistic IcebergSourceSplit with actual file paths.
+ *
+ * Actual data files are already deleted before return. Caller shouldn't attempt to read the
+ * data files. They are intended for serializer or enumerator unit test (without actual reading).
+ */
+ public static List<IcebergSourceSplit> createFileSplits(
+ TemporaryFolder temporaryFolder, int fileCount, int filesPerSplit) throws Exception {
+ File warehouseFile = temporaryFolder.newFolder();
+ Assert.assertTrue(warehouseFile.delete());
+ String warehouse = "file:" + warehouseFile;
+ org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
Review comment:
will fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r771893455
##########
File path: flink/v1.13/flink/src/test/java/org/apache/iceberg/flink/source/split/TestIcebergSourceSplitSerializer.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.junit.Assert;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+public class TestIcebergSourceSplitSerializer {
Review comment:
yes. will add
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#issuecomment-964812003
> Overall this looks really good to me. Would there be any benefit to us if we used [`FileSourceSplit` interface](https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/connector/file/src/FileSourceSplit.html) instead? Seems like that's a bit too general for our use case. And the SerDe is versioned anyway.
@kbendick the initial implementation does have `IcebergSourceSplit` extending from `FileSourceSplit` with the purpose of leveraging the vectorized readers from Flink (suggested by Jingsong). After thinking about it more with Iceberg's row delete filter, I changed the decision to the current state (no extending from FileSourceSplit).
When we are ready to support vectorized readers in Flink, we need to make sure they support delete filter properly. There is one open PR for Orc: https://github.com/apache/iceberg/pull/2566
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r766898835
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/Position.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+/**
+ * A mutable class that defines the read position
+ * <ul>
+ * <li>file offset in the list of files in a {@link CombinedScanTask}</li>
+ * <li>record offset within a file</li>
+ * </ul>
+ */
+@Internal
+public class Position implements Serializable {
Review comment:
`SplitPosition`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r766917245
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedBytesCache;
+
+ private IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedBytesCache() {
+ return serializedBytesCache;
+ }
+
+ void serializedBytesCache(byte[] cachedBytes) {
+ this.serializedBytesCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ IcebergSourceSplit split = (IcebergSourceSplit) o;
+ return Objects.equal(splitId(), split.splitId()) &&
+ Objects.equal(position, split.position());
Review comment:
Seems like you might want to just remove this and `hashCode` and use object identity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r766917801
##########
File path: flink/v1.13/flink/src/test/java/org/apache/iceberg/flink/source/split/SplitHelpers.java
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import org.apache.iceberg.BaseCombinedScanTask;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MockFileScanTask;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericAppenderHelper;
+import org.apache.iceberg.data.RandomGenericData;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.flink.TestFixtures;
+import org.apache.iceberg.flink.source.FlinkSplitPlanner;
+import org.apache.iceberg.flink.source.ScanContext;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.Assert;
+import org.junit.rules.TemporaryFolder;
+
+public class SplitHelpers {
+
+ private static final AtomicLong splitLengthIncrement = new AtomicLong();
+
+ private SplitHelpers() {
+
Review comment:
Nit: no need for a newline here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r769242277
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/Position.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+/**
+ * A mutable class that defines the read position
+ * <ul>
+ * <li>file offset in the list of files in a {@link CombinedScanTask}</li>
+ * <li>record offset within a file</li>
+ * </ul>
+ */
+@Internal
+public class Position implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ private int fileOffset;
+ private long recordOffset;
+
+ public Position(int fileOffset, long recordOffset) {
+ this.fileOffset = fileOffset;
+ this.recordOffset = recordOffset;
+ }
+
+ void advanceFile() {
+ this.fileOffset += 1;
+ this.recordOffset = 0L;
+ }
+
+ void advanceRecord() {
+ this.recordOffset += 1L;
+ }
+
+ public void update(int newFileOffset, long newRecordOffset) {
Review comment:
sure. will rename the method to `set`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r769242077
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/FlinkSplitPlanner.java
##########
@@ -22,33 +22,57 @@
import java.io.IOException;
import java.io.UncheckedIOException;
import java.util.List;
+import org.apache.flink.annotation.Internal;
import org.apache.iceberg.CombinedScanTask;
import org.apache.iceberg.Table;
import org.apache.iceberg.TableProperties;
import org.apache.iceberg.TableScan;
import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.flink.source.split.IcebergSourceSplit;
import org.apache.iceberg.io.CloseableIterable;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-class FlinkSplitGenerator {
- private FlinkSplitGenerator() {
+@Internal
+public class FlinkSplitPlanner {
+ private FlinkSplitPlanner() {
}
- static FlinkInputSplit[] createInputSplits(Table table, ScanContext context) {
- List<CombinedScanTask> tasks = tasks(table, context);
- FlinkInputSplit[] splits = new FlinkInputSplit[tasks.size()];
- for (int i = 0; i < tasks.size(); i++) {
- splits[i] = new FlinkInputSplit(i, tasks.get(i));
+ static FlinkInputSplit[] planInputSplits(Table table, ScanContext context) {
+ try (CloseableIterable<CombinedScanTask> tasksIterable = planTasks(table, context)) {
+ List<CombinedScanTask> tasks = Lists.newArrayList(tasksIterable);
+ FlinkInputSplit[] splits = new FlinkInputSplit[tasks.size()];
+ for (int i = 0; i < tasks.size(); i++) {
+ splits[i] = new FlinkInputSplit(i, tasks.get(i));
+ }
+ return splits;
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to process tasks iterable", e);
+ }
+ }
+
+ /**
+ * This returns splits for the FLIP-27 source
+ */
+ public static List<IcebergSourceSplit> planIcebergSourceSplits(Table table, ScanContext context) {
+ try (CloseableIterable<CombinedScanTask> tasksIterable = planTasks(table, context)) {
+ List<IcebergSourceSplit> splits = Lists.newArrayList();
+ tasksIterable.forEach(task -> splits.add(IcebergSourceSplit.fromCombinedScanTask(task)));
+ return splits;
Review comment:
sure. will change to this one-liner
```
return Lists.newArrayList(CloseableIterable.transform(tasksIterable,
task -> IcebergSourceSplit.fromCombinedScanTask(task)));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r771893448
##########
File path: flink/v1.13/flink/src/test/java/org/apache/iceberg/flink/source/split/SplitHelpers.java
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import org.apache.iceberg.BaseCombinedScanTask;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MockFileScanTask;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericAppenderHelper;
+import org.apache.iceberg.data.RandomGenericData;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.flink.TestFixtures;
+import org.apache.iceberg.flink.source.FlinkSplitPlanner;
+import org.apache.iceberg.flink.source.ScanContext;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.Assert;
+import org.junit.rules.TemporaryFolder;
+
+public class SplitHelpers {
+
+ private static final AtomicLong splitLengthIncrement = new AtomicLong();
+
+ private SplitHelpers() {
+
+ }
+
+ public static List<IcebergSourceSplit> createMockedSplits(int splitCount) {
+ List<IcebergSourceSplit> splits = new ArrayList<>();
+ for (int i = 0; i < splitCount; ++i) {
+ // make sure each task has a different length,
+ // as it is part of the splitId calculation.
+ // This way, we can make sure all generated splits have different splitIds
+ FileScanTask fileScanTask = new MockFileScanTask(1024 + splitLengthIncrement.incrementAndGet());
+ CombinedScanTask combinedScanTask = new BaseCombinedScanTask(fileScanTask);
+ splits.add(IcebergSourceSplit.fromCombinedScanTask(combinedScanTask));
+ }
+ return splits;
+ }
+
+ /**
+ * Unlike {@link SplitHelpers#createMockedSplits(int)} above, this is to generate some
+ * realistic IcebergSourceSplit with actual file paths.
+ *
+ * Actual data files are already deleted before return. Caller shouldn't attempt to read the
+ * data files. They are intended for serializer or enumerator unit test (without actual reading).
+ */
+ public static List<IcebergSourceSplit> createFileSplits(
+ TemporaryFolder temporaryFolder, int fileCount, int filesPerSplit) throws Exception {
Review comment:
+1. I am changing the method name to `createFileSplitsFromTransientHadoopTable` and also added some Javadoc. Open to other suggestions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r771989074
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedBytesCache;
+
+ private IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedBytesCache() {
+ return serializedBytesCache;
+ }
+
+ void serializedBytesCache(byte[] cachedBytes) {
+ this.serializedBytesCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ IcebergSourceSplit split = (IcebergSourceSplit) o;
+ return Objects.equal(splitId(), split.splitId()) &&
+ Objects.equal(position, split.position());
Review comment:
removed the `hashCode` and `equals`. Refactored other code to make this work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r771575200
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplitSerializer.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.IOException;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.util.InstantiationUtil;
+
+/**
+ * TODO: use Java serialization for now.
+ * Will switch to more stable serializer from
+ * <a href="https://github.com/apache/iceberg/issues/1698">issue-1698</a>.
+ */
+@Internal
+public class IcebergSourceSplitSerializer implements SimpleVersionedSerializer<IcebergSourceSplit> {
+ public static final IcebergSourceSplitSerializer INSTANCE = new IcebergSourceSplitSerializer();
+ private static final int VERSION = 1;
+
+ @Override
+ public int getVersion() {
+ return VERSION;
+ }
+
+ @Override
+ public byte[] serialize(IcebergSourceSplit split) throws IOException {
+ if (split.serializedBytesCache() == null) {
+ byte[] result = serializeV1(split);
+ split.serializedBytesCache(result);
Review comment:
will move the serialization inside the `IcebergSourceSplit` so that caching can be handled internally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx edited a comment on pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
openinx edited a comment on pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#issuecomment-969699867
Since we've enabled all the engine version's checkstyle check, let's rerun this travis CI again ! https://github.com/apache/iceberg/pull/3550
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r750212465
##########
File path: flink/v1.13/flink/src/test/java/org/apache/iceberg/flink/source/split/SplitHelpers.java
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import org.apache.iceberg.BaseCombinedScanTask;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MockFileScanTask;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericAppenderHelper;
+import org.apache.iceberg.data.RandomGenericData;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.flink.TestFixtures;
+import org.apache.iceberg.flink.source.FlinkSplitPlanner;
+import org.apache.iceberg.flink.source.ScanContext;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.Assert;
+import org.junit.rules.TemporaryFolder;
+
+public class SplitHelpers {
+
+ private static final AtomicLong splitLengthIncrement = new AtomicLong();
+
+ private SplitHelpers() {
+
+ }
+
+ public static List<IcebergSourceSplit> createMockedSplits(int splitCount) {
+ final List<IcebergSourceSplit> splits = new ArrayList<>();
+ for (int i = 0; i < splitCount; ++i) {
+ // make sure each task has a different length,
+ // as it is part of the splitId calculation.
+ // This way, we can make sure all generated splits have different splitIds
+ FileScanTask fileScanTask = new MockFileScanTask(1024 + splitLengthIncrement.incrementAndGet());
+ CombinedScanTask combinedScanTask = new BaseCombinedScanTask(fileScanTask);
+ splits.add(IcebergSourceSplit.fromCombinedScanTask(combinedScanTask));
+ }
+ return splits;
+ }
+
+ public static List<IcebergSourceSplit> createFileSplits(
+ TemporaryFolder temporaryFolder, int fileCount, int filesPerSplit) throws Exception {
+ final File warehouseFile = temporaryFolder.newFolder();
+ Assert.assertTrue(warehouseFile.delete());
+ final String warehouse = "file:" + warehouseFile;
+ org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
+ final HadoopCatalog catalog = new HadoopCatalog(hadoopConf, warehouse);
+ try {
+ final Table table = catalog.createTable(TestFixtures.TABLE_IDENTIFIER, TestFixtures.SCHEMA);
+ final GenericAppenderHelper dataAppender = new GenericAppenderHelper(
+ table, FileFormat.PARQUET, temporaryFolder);
+ for (int i = 0; i < fileCount; ++i) {
+ List<Record> records = RandomGenericData.generate(TestFixtures.SCHEMA, 2, i);
+ dataAppender.appendToTable(records);
+ }
+
+ final ScanContext scanContext = ScanContext.builder().build();
+ final List<IcebergSourceSplit> splits = FlinkSplitPlanner.planIcebergSourceSplits(table, scanContext);
+ return splits.stream()
+ .flatMap(split -> {
+ List<List<FileScanTask>> filesList = Lists.partition(new ArrayList<>(split.task().files()), filesPerSplit);
+ return filesList.stream()
+ .map(files -> new BaseCombinedScanTask(files))
+ .map(combinedScanTask -> IcebergSourceSplit.fromCombinedScanTask(combinedScanTask));
+ })
+ .collect(Collectors.toList());
+ } finally {
+ catalog.dropTable(TestFixtures.TABLE_IDENTIFIER);
Review comment:
As we will drop the table to remove data & metadata files in the end, could we read the `IcebergSourceSplit` in the outside method ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r761869611
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedFormCache;
+
+ public IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedFormCache() {
+ return serializedFormCache;
+ }
+
+ void serializedFormCache(byte[] cachedBytes) {
+ this.serializedFormCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ IcebergSourceSplit split = (IcebergSourceSplit) o;
+ return Objects.equal(splitId(), split.splitId()) &&
+ Objects.equal(position, split.position());
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hashCode(splitId());
Review comment:
I still don't think it valid to make the hash code build on top of the `FileScanTask#toString` because this [comment](https://github.com/apache/iceberg/pull/3501/files#r745462545).
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final CombinedScanTask task;
+ /**
Review comment:
I think this minor comment need to be addressed.
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/FlinkSplitPlanner.java
##########
@@ -22,33 +22,58 @@
import java.io.IOException;
import java.io.UncheckedIOException;
import java.util.List;
+import org.apache.flink.annotation.Internal;
import org.apache.iceberg.CombinedScanTask;
import org.apache.iceberg.Table;
import org.apache.iceberg.TableProperties;
import org.apache.iceberg.TableScan;
import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.flink.source.split.IcebergSourceSplit;
import org.apache.iceberg.io.CloseableIterable;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-class FlinkSplitGenerator {
- private FlinkSplitGenerator() {
+@Internal
+public class FlinkSplitPlanner {
+ private FlinkSplitPlanner() {
}
- static FlinkInputSplit[] createInputSplits(Table table, ScanContext context) {
- List<CombinedScanTask> tasks = tasks(table, context);
- FlinkInputSplit[] splits = new FlinkInputSplit[tasks.size()];
- for (int i = 0; i < tasks.size(); i++) {
- splits[i] = new FlinkInputSplit(i, tasks.get(i));
+ static FlinkInputSplit[] planInputSplits(Table table, ScanContext context) {
+ try (CloseableIterable<CombinedScanTask> tasksIterable = planTasks(table, context)) {
+ List<CombinedScanTask> tasks = Lists.newArrayList(tasksIterable);
+ FlinkInputSplit[] splits = new FlinkInputSplit[tasks.size()];
+ for (int i = 0; i < tasks.size(); i++) {
+ splits[i] = new FlinkInputSplit(i, tasks.get(i));
+ }
+ return splits;
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to process tasks iterable", e);
+ }
+ }
+
+ /**
+ * This returns splits for the FLIP-27 source
+ */
+ public static List<IcebergSourceSplit> planIcebergSourceSplits(
+ Table table, ScanContext context) {
+ try (CloseableIterable<CombinedScanTask> tasksIterable = planTasks(table, context)) {
+ List<IcebergSourceSplit> splits = Lists.newArrayList();
+ tasksIterable.forEach(task -> splits.add(IcebergSourceSplit.fromCombinedScanTask(task)));
+ return splits;
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to process task iterable: ", e);
}
- return splits;
}
- private static List<CombinedScanTask> tasks(Table table, ScanContext context) {
+ static CloseableIterable<CombinedScanTask> planTasks(Table table, ScanContext context) {
TableScan scan = table
.newScan()
.caseSensitive(context.caseSensitive())
.project(context.project());
+ if (context.includeColumnStats()) {
+ scan = scan.includeColumnStats();
+ }
Review comment:
@stevenzwu How is your feeling for this comment ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu edited a comment on pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu edited a comment on pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#issuecomment-964812003
> Would there be any benefit to us if we used [`FileSourceSplit` interface](https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/connector/file/src/FileSourceSplit.html) instead? Seems like that's a bit too general for our use case. And the SerDe is versioned anyway.
@kbendick the initial implementation does have `IcebergSourceSplit` extending from `FileSourceSplit` with the purpose of leveraging the vectorized readers from Flink (suggested by Jingsong). After thinking about it more with Iceberg's row delete filter, I changed the decision to the current state (no extending from FileSourceSplit).
When we are ready to support vectorized readers in Flink, we need to make sure they support delete filter properly. There is one open PR for Orc: https://github.com/apache/iceberg/pull/2566
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r772012894
##########
File path: flink/v1.13/flink/src/test/java/org/apache/iceberg/flink/source/split/SplitHelpers.java
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import org.apache.iceberg.BaseCombinedScanTask;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MockFileScanTask;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericAppenderHelper;
+import org.apache.iceberg.data.RandomGenericData;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.flink.TestFixtures;
+import org.apache.iceberg.flink.source.FlinkSplitPlanner;
+import org.apache.iceberg.flink.source.ScanContext;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.Assert;
+import org.junit.rules.TemporaryFolder;
+
+public class SplitHelpers {
+
+ private static final AtomicLong splitLengthIncrement = new AtomicLong();
+
+ private SplitHelpers() {
+
+ }
+
+ public static List<IcebergSourceSplit> createMockedSplits(int splitCount) {
+ final List<IcebergSourceSplit> splits = new ArrayList<>();
+ for (int i = 0; i < splitCount; ++i) {
+ // make sure each task has a different length,
+ // as it is part of the splitId calculation.
+ // This way, we can make sure all generated splits have different splitIds
+ FileScanTask fileScanTask = new MockFileScanTask(1024 + splitLengthIncrement.incrementAndGet());
+ CombinedScanTask combinedScanTask = new BaseCombinedScanTask(fileScanTask);
+ splits.add(IcebergSourceSplit.fromCombinedScanTask(combinedScanTask));
+ }
+ return splits;
+ }
+
+ public static List<IcebergSourceSplit> createFileSplits(
+ TemporaryFolder temporaryFolder, int fileCount, int filesPerSplit) throws Exception {
+ final File warehouseFile = temporaryFolder.newFolder();
+ Assert.assertTrue(warehouseFile.delete());
+ final String warehouse = "file:" + warehouseFile;
+ org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
+ final HadoopCatalog catalog = new HadoopCatalog(hadoopConf, warehouse);
+ try {
+ final Table table = catalog.createTable(TestFixtures.TABLE_IDENTIFIER, TestFixtures.SCHEMA);
+ final GenericAppenderHelper dataAppender = new GenericAppenderHelper(
+ table, FileFormat.PARQUET, temporaryFolder);
+ for (int i = 0; i < fileCount; ++i) {
+ List<Record> records = RandomGenericData.generate(TestFixtures.SCHEMA, 2, i);
+ dataAppender.appendToTable(records);
+ }
+
+ final ScanContext scanContext = ScanContext.builder().build();
+ final List<IcebergSourceSplit> splits = FlinkSplitPlanner.planIcebergSourceSplits(table, scanContext);
+ return splits.stream()
+ .flatMap(split -> {
+ List<List<FileScanTask>> filesList = Lists.partition(new ArrayList<>(split.task().files()), filesPerSplit);
+ return filesList.stream()
+ .map(files -> new BaseCombinedScanTask(files))
+ .map(combinedScanTask -> IcebergSourceSplit.fromCombinedScanTask(combinedScanTask));
+ })
+ .collect(Collectors.toList());
+ } finally {
+ catalog.dropTable(TestFixtures.TABLE_IDENTIFIER);
Review comment:
the method has been renamed and hopefully it won't be confusing. also updated javadoc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r766908567
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplitSerializer.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.IOException;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.util.InstantiationUtil;
+
+/**
+ * TODO: use Java serialization for now.
+ * Will switch to more stable serializer from
+ * <a href="https://github.com/apache/iceberg/issues/1698">issue-1698</a>.
+ */
+@Internal
+public class IcebergSourceSplitSerializer implements SimpleVersionedSerializer<IcebergSourceSplit> {
+ public static final IcebergSourceSplitSerializer INSTANCE = new IcebergSourceSplitSerializer();
+ private static final int VERSION = 1;
+
+ @Override
+ public int getVersion() {
+ return VERSION;
+ }
+
+ @Override
+ public byte[] serialize(IcebergSourceSplit split) throws IOException {
+ if (split.serializedBytesCache() == null) {
+ byte[] result = serializeV1(split);
+ split.serializedBytesCache(result);
+ }
+ return split.serializedBytesCache();
+ }
+
+ @Override
+ public IcebergSourceSplit deserialize(int version, byte[] serialized) throws IOException {
+ switch (version) {
+ case 1:
+ return deserializeV1(serialized);
+ default:
+ throw new IOException(String.format("Failed to deserialize IcebergSourceSplit. " +
+ "Encountered unsupported version: %d. Supported version are [1]", version));
+ }
+ }
+
+ @VisibleForTesting
+ byte[] serializeV1(IcebergSourceSplit split) throws IOException {
+ return InstantiationUtil.serializeObject(split);
+ }
+
+ @VisibleForTesting
+ IcebergSourceSplit deserializeV1(byte[] serialized) throws IOException {
+ try {
+ return InstantiationUtil.deserializeObject(serialized, getClass().getClassLoader());
Review comment:
Should the `ClassLoader` be `IcebergSourceSplit.class.getClassLoader()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r766938262
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/Position.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+/**
+ * A mutable class that defines the read position
+ * <ul>
+ * <li>file offset in the list of files in a {@link CombinedScanTask}</li>
+ * <li>record offset within a file</li>
+ * </ul>
+ */
+@Internal
+public class Position implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ private int fileOffset;
+ private long recordOffset;
+
+ public Position(int fileOffset, long recordOffset) {
+ this.fileOffset = fileOffset;
+ this.recordOffset = recordOffset;
+ }
+
+ void advanceFile() {
+ this.fileOffset += 1;
+ this.recordOffset = 0L;
+ }
+
+ void advanceRecord() {
+ this.recordOffset += 1L;
+ }
+
+ public void update(int newFileOffset, long newRecordOffset) {
+ this.fileOffset = newFileOffset;
+ this.recordOffset = newRecordOffset;
+ }
+
+ public int fileOffset() {
+ return fileOffset;
+ }
+
+ public long recordOffset() {
+ return recordOffset;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ Position that = (Position) o;
+ return Objects.equals(fileOffset, that.fileOffset) &&
+ Objects.equals(recordOffset, that.recordOffset);
Review comment:
These are primitives, so calling `equals(Object, Object)` will box both values just to do a more expensive check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r766919379
##########
File path: flink/v1.13/flink/src/test/java/org/apache/iceberg/flink/source/split/TestIcebergSourceSplitSerializer.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.junit.Assert;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+public class TestIcebergSourceSplitSerializer {
Review comment:
Probably want a test for calling `advance` on the position and then serializing the split.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r767392670
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/Position.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+/**
+ * A mutable class that defines the read position
+ * <ul>
+ * <li>file offset in the list of files in a {@link CombinedScanTask}</li>
+ * <li>record offset within a file</li>
+ * </ul>
+ */
+@Internal
+public class Position implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ private int fileOffset;
+ private long recordOffset;
+
+ public Position(int fileOffset, long recordOffset) {
+ this.fileOffset = fileOffset;
+ this.recordOffset = recordOffset;
+ }
+
+ void advanceFile() {
+ this.fileOffset += 1;
+ this.recordOffset = 0L;
+ }
+
+ void advanceRecord() {
+ this.recordOffset += 1L;
+ }
+
+ public void update(int newFileOffset, long newRecordOffset) {
+ this.fileOffset = newFileOffset;
+ this.recordOffset = newRecordOffset;
+ }
+
+ public int fileOffset() {
+ return fileOffset;
+ }
+
+ public long recordOffset() {
+ return recordOffset;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ Position that = (Position) o;
+ return Objects.equals(fileOffset, that.fileOffset) &&
+ Objects.equals(recordOffset, that.recordOffset);
Review comment:
Oh good call hadn’t considered the boxing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r771893886
##########
File path: flink/v1.13/flink/src/test/java/org/apache/iceberg/flink/source/split/TestIcebergSourceSplitSerializer.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.junit.Assert;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+public class TestIcebergSourceSplitSerializer {
+
+ @ClassRule
+ public static final TemporaryFolder TEMPORARY_FOLDER = new TemporaryFolder();
+
+ private final IcebergSourceSplitSerializer serializer = IcebergSourceSplitSerializer.INSTANCE;
+
+ @Test
+ public void testLatestVersion() throws Exception {
+ serializeAndDeserialize(1, 1);
+ serializeAndDeserialize(10, 2);
+ }
+
+ private void serializeAndDeserialize(int splitCount, int filesPerSplit) throws Exception {
+ List<IcebergSourceSplit> splits = SplitHelpers.createFileSplits(TEMPORARY_FOLDER, splitCount, filesPerSplit);
+ for (IcebergSourceSplit split : splits) {
+ byte[] result = serializer.serialize(split);
+ IcebergSourceSplit deserialized = serializer.deserialize(serializer.getVersion(), result);
+ Assert.assertEquals(split, deserialized);
+
+ byte[] cachedResult = serializer.serialize(split);
+ Assert.assertSame(result, cachedResult);
+ IcebergSourceSplit deserialized2 = serializer.deserialize(serializer.getVersion(), cachedResult);
+ Assert.assertEquals(split, deserialized2);
+ }
+ }
+
+ @Test
+ public void testV1() throws Exception {
+ serializeAndDeserializeV1(1, 1);
+ serializeAndDeserializeV1(10, 2);
+ }
+
+ private void serializeAndDeserializeV1(int splitCount, int filesPerSplit) throws Exception {
+ List<IcebergSourceSplit> splits = SplitHelpers.createFileSplits(TEMPORARY_FOLDER, splitCount, filesPerSplit);
+ for (IcebergSourceSplit split : splits) {
+ byte[] result = serializer.serializeV1(split);
+ IcebergSourceSplit deserialized = serializer.deserializeV1(result);
+ Assert.assertEquals(split, deserialized);
+ }
+ }
+
+ @Test
+ public void testCheckpointedPosition() throws Exception {
+ AtomicInteger index = new AtomicInteger();
+ List<IcebergSourceSplit> splits = SplitHelpers.createFileSplits(TEMPORARY_FOLDER, 10, 2).stream()
+ .map(split -> {
+ IcebergSourceSplit result;
+ if (index.get() % 2 == 0) {
+ result = IcebergSourceSplit.fromCombinedScanTask(split.task(), index.get(), index.get());
Review comment:
Added the serializer test after changing the position.
`advanceFile` and `advanceRecord` are only used by `DataIterator` class. They aren't used by `IcebergSourceSplit`, which only use `set(int newFileOffset, long newRecordOffset)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r771575764
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedBytesCache;
+
+ private IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedBytesCache() {
+ return serializedBytesCache;
+ }
+
+ void serializedBytesCache(byte[] cachedBytes) {
+ this.serializedBytesCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
Review comment:
correct. this is a bug. will fix and update unit test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r771575552
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedBytesCache;
+
+ private IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
Review comment:
yes. this is actually a bug. Updated unit test to cover the case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r772013094
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final CombinedScanTask task;
+ /**
Review comment:
changed to `//` comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r750211046
##########
File path: flink/v1.13/flink/src/test/java/org/apache/iceberg/flink/source/split/SplitHelpers.java
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import org.apache.iceberg.BaseCombinedScanTask;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MockFileScanTask;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericAppenderHelper;
+import org.apache.iceberg.data.RandomGenericData;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.flink.TestFixtures;
+import org.apache.iceberg.flink.source.FlinkSplitPlanner;
+import org.apache.iceberg.flink.source.ScanContext;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.Assert;
+import org.junit.rules.TemporaryFolder;
+
+public class SplitHelpers {
+
+ private static final AtomicLong splitLengthIncrement = new AtomicLong();
+
+ private SplitHelpers() {
+
+ }
+
+ public static List<IcebergSourceSplit> createMockedSplits(int splitCount) {
+ final List<IcebergSourceSplit> splits = new ArrayList<>();
+ for (int i = 0; i < splitCount; ++i) {
+ // make sure each task has a different length,
+ // as it is part of the splitId calculation.
+ // This way, we can make sure all generated splits have different splitIds
+ FileScanTask fileScanTask = new MockFileScanTask(1024 + splitLengthIncrement.incrementAndGet());
+ CombinedScanTask combinedScanTask = new BaseCombinedScanTask(fileScanTask);
+ splits.add(IcebergSourceSplit.fromCombinedScanTask(combinedScanTask));
+ }
+ return splits;
+ }
+
+ public static List<IcebergSourceSplit> createFileSplits(
+ TemporaryFolder temporaryFolder, int fileCount, int filesPerSplit) throws Exception {
+ final File warehouseFile = temporaryFolder.newFolder();
Review comment:
Nit: it is recommended to remove all those unnecessary `final` modifiers to keep consistent with iceberg coding styles in my mind.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r766919048
##########
File path: flink/v1.13/flink/src/test/java/org/apache/iceberg/flink/source/split/SplitHelpers.java
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import org.apache.iceberg.BaseCombinedScanTask;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MockFileScanTask;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericAppenderHelper;
+import org.apache.iceberg.data.RandomGenericData;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.flink.TestFixtures;
+import org.apache.iceberg.flink.source.FlinkSplitPlanner;
+import org.apache.iceberg.flink.source.ScanContext;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.Assert;
+import org.junit.rules.TemporaryFolder;
+
+public class SplitHelpers {
+
+ private static final AtomicLong splitLengthIncrement = new AtomicLong();
+
+ private SplitHelpers() {
+
+ }
+
+ public static List<IcebergSourceSplit> createMockedSplits(int splitCount) {
+ List<IcebergSourceSplit> splits = new ArrayList<>();
+ for (int i = 0; i < splitCount; ++i) {
+ // make sure each task has a different length,
+ // as it is part of the splitId calculation.
+ // This way, we can make sure all generated splits have different splitIds
+ FileScanTask fileScanTask = new MockFileScanTask(1024 + splitLengthIncrement.incrementAndGet());
+ CombinedScanTask combinedScanTask = new BaseCombinedScanTask(fileScanTask);
+ splits.add(IcebergSourceSplit.fromCombinedScanTask(combinedScanTask));
+ }
+ return splits;
+ }
+
+ /**
+ * Unlike {@link SplitHelpers#createMockedSplits(int)} above, this is to generate some
+ * realistic IcebergSourceSplit with actual file paths.
+ *
+ * Actual data files are already deleted before return. Caller shouldn't attempt to read the
+ * data files. They are intended for serializer or enumerator unit test (without actual reading).
+ */
+ public static List<IcebergSourceSplit> createFileSplits(
+ TemporaryFolder temporaryFolder, int fileCount, int filesPerSplit) throws Exception {
Review comment:
Is there a better name that shows that this is creating a table, writing data, and then planning a scan?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r766915694
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedBytesCache;
+
+ private IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedBytesCache() {
+ return serializedBytesCache;
+ }
+
+ void serializedBytesCache(byte[] cachedBytes) {
+ this.serializedBytesCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ IcebergSourceSplit split = (IcebergSourceSplit) o;
+ return Objects.equal(splitId(), split.splitId()) &&
+ Objects.equal(position, split.position());
Review comment:
The `equals` and `hashCode` implementations are not consistent because `equals` uses the position, but `hashCode` ignores it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r766920975
##########
File path: flink/v1.13/flink/src/test/java/org/apache/iceberg/flink/source/split/TestIcebergSourceSplitSerializer.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+import org.junit.Assert;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+public class TestIcebergSourceSplitSerializer {
+
+ @ClassRule
+ public static final TemporaryFolder TEMPORARY_FOLDER = new TemporaryFolder();
+
+ private final IcebergSourceSplitSerializer serializer = IcebergSourceSplitSerializer.INSTANCE;
+
+ @Test
+ public void testLatestVersion() throws Exception {
+ serializeAndDeserialize(1, 1);
+ serializeAndDeserialize(10, 2);
+ }
+
+ private void serializeAndDeserialize(int splitCount, int filesPerSplit) throws Exception {
+ List<IcebergSourceSplit> splits = SplitHelpers.createFileSplits(TEMPORARY_FOLDER, splitCount, filesPerSplit);
+ for (IcebergSourceSplit split : splits) {
+ byte[] result = serializer.serialize(split);
+ IcebergSourceSplit deserialized = serializer.deserialize(serializer.getVersion(), result);
+ Assert.assertEquals(split, deserialized);
+
+ byte[] cachedResult = serializer.serialize(split);
+ Assert.assertSame(result, cachedResult);
+ IcebergSourceSplit deserialized2 = serializer.deserialize(serializer.getVersion(), cachedResult);
+ Assert.assertEquals(split, deserialized2);
+ }
+ }
+
+ @Test
+ public void testV1() throws Exception {
+ serializeAndDeserializeV1(1, 1);
+ serializeAndDeserializeV1(10, 2);
+ }
+
+ private void serializeAndDeserializeV1(int splitCount, int filesPerSplit) throws Exception {
+ List<IcebergSourceSplit> splits = SplitHelpers.createFileSplits(TEMPORARY_FOLDER, splitCount, filesPerSplit);
+ for (IcebergSourceSplit split : splits) {
+ byte[] result = serializer.serializeV1(split);
+ IcebergSourceSplit deserialized = serializer.deserializeV1(result);
+ Assert.assertEquals(split, deserialized);
+ }
+ }
+
+ @Test
+ public void testCheckpointedPosition() throws Exception {
+ AtomicInteger index = new AtomicInteger();
+ List<IcebergSourceSplit> splits = SplitHelpers.createFileSplits(TEMPORARY_FOLDER, 10, 2).stream()
+ .map(split -> {
+ IcebergSourceSplit result;
+ if (index.get() % 2 == 0) {
+ result = IcebergSourceSplit.fromCombinedScanTask(split.task(), index.get(), index.get());
Review comment:
Before this, call `serializer.serialize(...)` to test that serialization is correct after changing the position.
Some of these should also call `advanceFile()` and `advanceRow()` as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r766911347
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedBytesCache;
+
+ private IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedBytesCache() {
+ return serializedBytesCache;
+ }
+
+ void serializedBytesCache(byte[] cachedBytes) {
+ this.serializedBytesCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
Review comment:
Invalidate serialized bytes here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r766906148
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplitSerializer.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.IOException;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.util.InstantiationUtil;
+
+/**
+ * TODO: use Java serialization for now.
+ * Will switch to more stable serializer from
+ * <a href="https://github.com/apache/iceberg/issues/1698">issue-1698</a>.
+ */
+@Internal
+public class IcebergSourceSplitSerializer implements SimpleVersionedSerializer<IcebergSourceSplit> {
+ public static final IcebergSourceSplitSerializer INSTANCE = new IcebergSourceSplitSerializer();
+ private static final int VERSION = 1;
+
+ @Override
+ public int getVersion() {
+ return VERSION;
+ }
+
+ @Override
+ public byte[] serialize(IcebergSourceSplit split) throws IOException {
+ if (split.serializedBytesCache() == null) {
+ byte[] result = serializeV1(split);
+ split.serializedBytesCache(result);
Review comment:
Can we move serialization inside the split? We could then call `split.serializeV1()` and it could cache internally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#issuecomment-968344249
@openinx can you take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
openinx commented on pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#issuecomment-969699867
Since we've enabled all the engine version's checkstyle check, let's rerun this travis CI again !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r769242852
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/Position.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+/**
+ * A mutable class that defines the read position
+ * <ul>
+ * <li>file offset in the list of files in a {@link CombinedScanTask}</li>
+ * <li>record offset within a file</li>
+ * </ul>
+ */
+@Internal
+public class Position implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ private int fileOffset;
+ private long recordOffset;
+
+ public Position(int fileOffset, long recordOffset) {
+ this.fileOffset = fileOffset;
+ this.recordOffset = recordOffset;
+ }
+
+ void advanceFile() {
+ this.fileOffset += 1;
+ this.recordOffset = 0L;
+ }
+
+ void advanceRecord() {
+ this.recordOffset += 1L;
+ }
+
+ public void update(int newFileOffset, long newRecordOffset) {
+ this.fileOffset = newFileOffset;
+ this.recordOffset = newRecordOffset;
+ }
+
+ public int fileOffset() {
+ return fileOffset;
+ }
+
+ public long recordOffset() {
+ return recordOffset;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ Position that = (Position) o;
+ return Objects.equals(fileOffset, that.fileOffset) &&
+ Objects.equals(recordOffset, that.recordOffset);
Review comment:
will change to `==`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r771892927
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedBytesCache;
+
+ private IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedBytesCache() {
+ return serializedBytesCache;
+ }
+
+ void serializedBytesCache(byte[] cachedBytes) {
+ this.serializedBytesCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ IcebergSourceSplit split = (IcebergSourceSplit) o;
+ return Objects.equal(splitId(), split.splitId()) &&
+ Objects.equal(position, split.position());
Review comment:
agree with the inconsistent problem. let me look into this. We do have places where those two overrides are used.
e.g.
```
public class IcebergEnumeratorState implements Serializable {
private final Map<IcebergSourceSplit, IcebergSourceSplitStatus> pendingSplits;
}
```
Also a few unit tests rely on the `equals` to verify deserialized objects are the same as the original one. I am looking to replace those with a `Comparator` impl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r771892927
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/split/IcebergSourceSplit.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.flink.source.Position;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Objects;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+
+@Internal
+public class IcebergSourceSplit implements SourceSplit, Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final CombinedScanTask task;
+ /**
+ * Position field is mutable
+ */
+ @Nullable
+ private final Position position;
+
+ /**
+ * The splits are frequently serialized into checkpoints.
+ * Caching the byte representation makes repeated serialization cheap.
+ */
+ @Nullable
+ private transient byte[] serializedBytesCache;
+
+ private IcebergSourceSplit(CombinedScanTask task, Position position) {
+ this.task = task;
+ this.position = position;
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(CombinedScanTask combinedScanTask) {
+ return fromCombinedScanTask(combinedScanTask, 0, 0L);
+ }
+
+ public static IcebergSourceSplit fromCombinedScanTask(
+ CombinedScanTask combinedScanTask, int fileOffset, long recordOffset) {
+ return new IcebergSourceSplit(combinedScanTask, new Position(fileOffset, recordOffset));
+ }
+
+ public CombinedScanTask task() {
+ return task;
+ }
+
+ public Position position() {
+ return position;
+ }
+
+ byte[] serializedBytesCache() {
+ return serializedBytesCache;
+ }
+
+ void serializedBytesCache(byte[] cachedBytes) {
+ this.serializedBytesCache = cachedBytes;
+ }
+
+ @Override
+ public String splitId() {
+ return MoreObjects.toStringHelper(this)
+ .add("files", toString(task.files()))
+ .toString();
+ }
+
+ public void updatePosition(int newFileOffset, long newRecordOffset) {
+ position.update(newFileOffset, newRecordOffset);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ IcebergSourceSplit split = (IcebergSourceSplit) o;
+ return Objects.equal(splitId(), split.splitId()) &&
+ Objects.equal(position, split.position());
Review comment:
Yeah. Looking at the usage again, we should just remove the `equals` and `hashCode` implementations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r771893016
##########
File path: flink/v1.13/flink/src/test/java/org/apache/iceberg/flink/source/split/SplitHelpers.java
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source.split;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+import org.apache.iceberg.BaseCombinedScanTask;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.MockFileScanTask;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericAppenderHelper;
+import org.apache.iceberg.data.RandomGenericData;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.flink.TestFixtures;
+import org.apache.iceberg.flink.source.FlinkSplitPlanner;
+import org.apache.iceberg.flink.source.ScanContext;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.Assert;
+import org.junit.rules.TemporaryFolder;
+
+public class SplitHelpers {
+
+ private static final AtomicLong splitLengthIncrement = new AtomicLong();
+
+ private SplitHelpers() {
+
Review comment:
will fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3501: Flink: FLIP-27 Iceberg source split
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3501:
URL: https://github.com/apache/iceberg/pull/3501#discussion_r771111487
##########
File path: flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/Position.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink.source;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.flink.annotation.Internal;
+import org.apache.iceberg.CombinedScanTask;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+
+/**
+ * A mutable class that defines the read position
+ * <ul>
+ * <li>file offset in the list of files in a {@link CombinedScanTask}</li>
+ * <li>record offset within a file</li>
+ * </ul>
+ */
+@Internal
+public class Position implements Serializable {
Review comment:
will rename the class to `SplitPosition`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org