You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/11/16 22:39:29 UTC
[GitHub] [iceberg] aokolnychyi opened a new pull request #1776: Spark: Refactor Spark writes into separate classes
aokolnychyi opened a new pull request #1776:
URL: https://github.com/apache/iceberg/pull/1776
This PR refactors our Spark writes into separate classes so that we can add more writes with other commit functions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #1776: Spark: Refactor Spark writes into separate classes (V2)
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #1776:
URL: https://github.com/apache/iceberg/pull/1776#discussion_r524817542
##########
File path: spark3/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java
##########
@@ -236,9 +201,171 @@ protected Table table() {
return ImmutableList.of();
}
- @Override
- public String toString() {
- return String.format("IcebergWrite(table=%s, format=%s)", table, format);
+ private abstract class BaseBatchWrite implements BatchWrite {
+ @Override
+ public DataWriterFactory createBatchWriterFactory(PhysicalWriteInfo info) {
+ return createWriterFactory();
+ }
+
+ @Override
+ public void abort(WriterCommitMessage[] messages) {
+ SparkWrite.this.abort(messages);
+ }
+
+ @Override
+ public String toString() {
+ return String.format("IcebergBatchWrite(table=%s, format=%s)", table, format);
+ }
+ }
+
+ private class BatchAppend extends BaseBatchWrite {
+ @Override
+ public void commit(WriterCommitMessage[] messages) {
+ AppendFiles append = table.newAppend();
+
+ int numFiles = 0;
+ for (DataFile file : files(messages)) {
+ numFiles += 1;
+ append.appendFile(file);
+ }
+
+ commitOperation(append, String.format("append with %d new data files", numFiles));
+ }
+ }
+
+ private class DynamicOverwrite extends BaseBatchWrite {
+ @Override
+ public void commit(WriterCommitMessage[] messages) {
+ ReplacePartitions dynamicOverwrite = table.newReplacePartitions();
+
+ int numFiles = 0;
+ for (DataFile file : files(messages)) {
+ numFiles += 1;
+ dynamicOverwrite.addFile(file);
+ }
+
+ commitOperation(dynamicOverwrite, String.format("dynamic partition overwrite with %d new data files", numFiles));
+ }
+ }
+
+ private class OverwriteByFilter extends BaseBatchWrite {
+ private final Expression overwriteExpr;
+
+ private OverwriteByFilter(Expression overwriteExpr) {
+ this.overwriteExpr = overwriteExpr;
+ }
+
+ @Override
+ public void commit(WriterCommitMessage[] messages) {
+ OverwriteFiles overwriteFiles = table.newOverwrite();
+ overwriteFiles.overwriteByRowFilter(overwriteExpr);
+
+ int numFiles = 0;
+ for (DataFile file : files(messages)) {
+ numFiles += 1;
+ overwriteFiles.addFile(file);
+ }
+
+ String commitMsg = String.format("overwrite by filter %s with %d new data files", overwriteExpr, numFiles);
+ commitOperation(overwriteFiles, commitMsg);
+ }
+ }
+
+ private abstract class BaseStreamingWrite implements StreamingWrite {
+ private static final String QUERY_ID_PROPERTY = "spark.sql.streaming.queryId";
+ private static final String EPOCH_ID_PROPERTY = "spark.sql.streaming.epochId";
+
+ protected abstract String mode();
+
+ @Override
+ public StreamingDataWriterFactory createStreamingWriterFactory(PhysicalWriteInfo info) {
+ return createWriterFactory();
+ }
+
+ @Override
+ public final void commit(long epochId, WriterCommitMessage[] messages) {
+ LOG.info("Committing epoch {} for query {} in {} mode", epochId, queryId, mode());
+
+ table.refresh();
+ Long lastCommittedEpochId = getLastCommittedEpochId();
+ if (lastCommittedEpochId != null && epochId <= lastCommittedEpochId) {
+ LOG.info("Skipping epoch {} for query {} as it was already committed", epochId, queryId);
+ return;
+ }
+
+ doCommit(epochId, messages);
+ }
+
+ protected abstract void doCommit(long epochId, WriterCommitMessage[] messages);
+
+ protected <T> void commit(SnapshotUpdate<T> snapshotUpdate, long epochId, String description) {
+ snapshotUpdate.set(QUERY_ID_PROPERTY, queryId);
+ snapshotUpdate.set(EPOCH_ID_PROPERTY, Long.toString(epochId));
+ commitOperation(snapshotUpdate, description);
+ }
+
+ private Long getLastCommittedEpochId() {
Review comment:
I kept the old name but I agree `find` is better. Let me update that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #1776: Spark: Refactor Spark writes into separate classes (V2)
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #1776:
URL: https://github.com/apache/iceberg/pull/1776#discussion_r524696424
##########
File path: spark3/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java
##########
@@ -72,52 +75,67 @@
import static org.apache.iceberg.TableProperties.WRITE_TARGET_FILE_SIZE_BYTES;
import static org.apache.iceberg.TableProperties.WRITE_TARGET_FILE_SIZE_BYTES_DEFAULT;
-class SparkBatchWrite implements BatchWrite {
- private static final Logger LOG = LoggerFactory.getLogger(SparkBatchWrite.class);
+class SparkWrite {
+ private static final Logger LOG = LoggerFactory.getLogger(SparkWrite.class);
private final Table table;
+ private final String queryId;
private final FileFormat format;
private final Broadcast<FileIO> io;
private final Broadcast<EncryptionManager> encryptionManager;
- private final boolean overwriteDynamic;
Review comment:
`SparkWrite` contains only common things now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi merged pull request #1776: Spark: Refactor Spark writes into separate classes (V2)
Posted by GitBox <gi...@apache.org>.
aokolnychyi merged pull request #1776:
URL: https://github.com/apache/iceberg/pull/1776
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1776: Spark: Refactor Spark writes into separate classes (V2)
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1776:
URL: https://github.com/apache/iceberg/pull/1776#discussion_r524791992
##########
File path: spark3/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java
##########
@@ -236,9 +201,171 @@ protected Table table() {
return ImmutableList.of();
}
- @Override
- public String toString() {
- return String.format("IcebergWrite(table=%s, format=%s)", table, format);
+ private abstract class BaseBatchWrite implements BatchWrite {
+ @Override
+ public DataWriterFactory createBatchWriterFactory(PhysicalWriteInfo info) {
+ return createWriterFactory();
+ }
+
+ @Override
+ public void abort(WriterCommitMessage[] messages) {
+ SparkWrite.this.abort(messages);
+ }
+
+ @Override
+ public String toString() {
+ return String.format("IcebergBatchWrite(table=%s, format=%s)", table, format);
+ }
+ }
+
+ private class BatchAppend extends BaseBatchWrite {
+ @Override
+ public void commit(WriterCommitMessage[] messages) {
+ AppendFiles append = table.newAppend();
+
+ int numFiles = 0;
+ for (DataFile file : files(messages)) {
+ numFiles += 1;
+ append.appendFile(file);
+ }
+
+ commitOperation(append, String.format("append with %d new data files", numFiles));
+ }
+ }
+
+ private class DynamicOverwrite extends BaseBatchWrite {
+ @Override
+ public void commit(WriterCommitMessage[] messages) {
+ ReplacePartitions dynamicOverwrite = table.newReplacePartitions();
+
+ int numFiles = 0;
+ for (DataFile file : files(messages)) {
+ numFiles += 1;
+ dynamicOverwrite.addFile(file);
+ }
+
+ commitOperation(dynamicOverwrite, String.format("dynamic partition overwrite with %d new data files", numFiles));
+ }
+ }
+
+ private class OverwriteByFilter extends BaseBatchWrite {
+ private final Expression overwriteExpr;
+
+ private OverwriteByFilter(Expression overwriteExpr) {
+ this.overwriteExpr = overwriteExpr;
+ }
+
+ @Override
+ public void commit(WriterCommitMessage[] messages) {
+ OverwriteFiles overwriteFiles = table.newOverwrite();
+ overwriteFiles.overwriteByRowFilter(overwriteExpr);
+
+ int numFiles = 0;
+ for (DataFile file : files(messages)) {
+ numFiles += 1;
+ overwriteFiles.addFile(file);
+ }
+
+ String commitMsg = String.format("overwrite by filter %s with %d new data files", overwriteExpr, numFiles);
+ commitOperation(overwriteFiles, commitMsg);
+ }
+ }
+
+ private abstract class BaseStreamingWrite implements StreamingWrite {
+ private static final String QUERY_ID_PROPERTY = "spark.sql.streaming.queryId";
+ private static final String EPOCH_ID_PROPERTY = "spark.sql.streaming.epochId";
+
+ protected abstract String mode();
+
+ @Override
+ public StreamingDataWriterFactory createStreamingWriterFactory(PhysicalWriteInfo info) {
+ return createWriterFactory();
+ }
+
+ @Override
+ public final void commit(long epochId, WriterCommitMessage[] messages) {
+ LOG.info("Committing epoch {} for query {} in {} mode", epochId, queryId, mode());
+
+ table.refresh();
+ Long lastCommittedEpochId = getLastCommittedEpochId();
+ if (lastCommittedEpochId != null && epochId <= lastCommittedEpochId) {
+ LOG.info("Skipping epoch {} for query {} as it was already committed", epochId, queryId);
+ return;
+ }
+
+ doCommit(epochId, messages);
+ }
+
+ protected abstract void doCommit(long epochId, WriterCommitMessage[] messages);
+
+ protected <T> void commit(SnapshotUpdate<T> snapshotUpdate, long epochId, String description) {
+ snapshotUpdate.set(QUERY_ID_PROPERTY, queryId);
+ snapshotUpdate.set(EPOCH_ID_PROPERTY, Long.toString(epochId));
+ commitOperation(snapshotUpdate, description);
+ }
+
+ private Long getLastCommittedEpochId() {
Review comment:
Since this searches through old snapshots, what about `find` instead of `get` here? That signals that the operation is doing more than just fetching a pre-computed value.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #1776: Spark: Refactor Spark writes into separate classes (V2)
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #1776:
URL: https://github.com/apache/iceberg/pull/1776#discussion_r524697102
##########
File path: spark3/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java
##########
@@ -170,45 +177,7 @@ protected void commitOperation(SnapshotUpdate<?> operation, int numFiles, String
LOG.info("Committed in {} ms", duration);
}
- private void append(WriterCommitMessage[] messages) {
Review comment:
This logic moved to inner classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #1776: Spark: Refactor Spark writes into separate classes (V2)
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #1776:
URL: https://github.com/apache/iceberg/pull/1776#discussion_r524701978
##########
File path: spark3/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java
##########
@@ -72,52 +75,67 @@
import static org.apache.iceberg.TableProperties.WRITE_TARGET_FILE_SIZE_BYTES;
import static org.apache.iceberg.TableProperties.WRITE_TARGET_FILE_SIZE_BYTES_DEFAULT;
-class SparkBatchWrite implements BatchWrite {
- private static final Logger LOG = LoggerFactory.getLogger(SparkBatchWrite.class);
+class SparkWrite {
Review comment:
In the future, Spark may have a write abstraction so we will make this class extend `Write`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1776: Spark: Refactor Spark writes into separate classes (V2)
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1776:
URL: https://github.com/apache/iceberg/pull/1776#issuecomment-728475624
+1
I think this is much better and I like that it removes the separate subclass for streaming.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #1776: Spark: Refactor Spark writes into separate classes (V2)
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #1776:
URL: https://github.com/apache/iceberg/pull/1776#discussion_r524948522
##########
File path: spark3/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java
##########
@@ -236,9 +201,171 @@ protected Table table() {
return ImmutableList.of();
}
- @Override
- public String toString() {
- return String.format("IcebergWrite(table=%s, format=%s)", table, format);
+ private abstract class BaseBatchWrite implements BatchWrite {
+ @Override
+ public DataWriterFactory createBatchWriterFactory(PhysicalWriteInfo info) {
+ return createWriterFactory();
+ }
+
+ @Override
+ public void abort(WriterCommitMessage[] messages) {
+ SparkWrite.this.abort(messages);
+ }
+
+ @Override
+ public String toString() {
+ return String.format("IcebergBatchWrite(table=%s, format=%s)", table, format);
+ }
+ }
+
+ private class BatchAppend extends BaseBatchWrite {
+ @Override
+ public void commit(WriterCommitMessage[] messages) {
+ AppendFiles append = table.newAppend();
+
+ int numFiles = 0;
+ for (DataFile file : files(messages)) {
+ numFiles += 1;
+ append.appendFile(file);
+ }
+
+ commitOperation(append, String.format("append with %d new data files", numFiles));
+ }
+ }
+
+ private class DynamicOverwrite extends BaseBatchWrite {
+ @Override
+ public void commit(WriterCommitMessage[] messages) {
+ ReplacePartitions dynamicOverwrite = table.newReplacePartitions();
+
+ int numFiles = 0;
+ for (DataFile file : files(messages)) {
+ numFiles += 1;
+ dynamicOverwrite.addFile(file);
+ }
+
+ commitOperation(dynamicOverwrite, String.format("dynamic partition overwrite with %d new data files", numFiles));
+ }
+ }
+
+ private class OverwriteByFilter extends BaseBatchWrite {
+ private final Expression overwriteExpr;
+
+ private OverwriteByFilter(Expression overwriteExpr) {
+ this.overwriteExpr = overwriteExpr;
+ }
+
+ @Override
+ public void commit(WriterCommitMessage[] messages) {
+ OverwriteFiles overwriteFiles = table.newOverwrite();
+ overwriteFiles.overwriteByRowFilter(overwriteExpr);
+
+ int numFiles = 0;
+ for (DataFile file : files(messages)) {
+ numFiles += 1;
+ overwriteFiles.addFile(file);
+ }
+
+ String commitMsg = String.format("overwrite by filter %s with %d new data files", overwriteExpr, numFiles);
+ commitOperation(overwriteFiles, commitMsg);
+ }
+ }
+
+ private abstract class BaseStreamingWrite implements StreamingWrite {
+ private static final String QUERY_ID_PROPERTY = "spark.sql.streaming.queryId";
+ private static final String EPOCH_ID_PROPERTY = "spark.sql.streaming.epochId";
+
+ protected abstract String mode();
+
+ @Override
+ public StreamingDataWriterFactory createStreamingWriterFactory(PhysicalWriteInfo info) {
+ return createWriterFactory();
+ }
+
+ @Override
+ public final void commit(long epochId, WriterCommitMessage[] messages) {
+ LOG.info("Committing epoch {} for query {} in {} mode", epochId, queryId, mode());
+
+ table.refresh();
+ Long lastCommittedEpochId = getLastCommittedEpochId();
+ if (lastCommittedEpochId != null && epochId <= lastCommittedEpochId) {
+ LOG.info("Skipping epoch {} for query {} as it was already committed", epochId, queryId);
+ return;
+ }
+
+ doCommit(epochId, messages);
+ }
+
+ protected abstract void doCommit(long epochId, WriterCommitMessage[] messages);
+
+ protected <T> void commit(SnapshotUpdate<T> snapshotUpdate, long epochId, String description) {
+ snapshotUpdate.set(QUERY_ID_PROPERTY, queryId);
+ snapshotUpdate.set(EPOCH_ID_PROPERTY, Long.toString(epochId));
+ commitOperation(snapshotUpdate, description);
+ }
+
+ private Long getLastCommittedEpochId() {
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #1776: Spark: Refactor Spark writes into separate classes (V2)
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #1776:
URL: https://github.com/apache/iceberg/pull/1776#discussion_r524697662
##########
File path: spark3/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java
##########
@@ -236,9 +201,171 @@ protected Table table() {
return ImmutableList.of();
}
- @Override
- public String toString() {
Review comment:
This logic moved to base batch and streaming write classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org