You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/12/22 04:01:08 UTC
[GitHub] [iceberg] amogh-jahagirdar opened a new pull request, #6480: Spark: Fail streaming planning when snapshot not found
amogh-jahagirdar opened a new pull request, #6480:
URL: https://github.com/apache/iceberg/pull/6480
Fixing error handling for https://github.com/apache/iceberg/issues/6388.
Based on the stack trace the following sequence of events seems plausible.
1.) [The snapshot ID for current offset no longer exists (my hunch is due to expiration)](https://github.com/apache/iceberg/blob/master/spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java#L210). So table.snapshot(currentOffset.snapshotId()) returns null.
2.) Then planning throws an unclear [NPE](https://github.com/apache/iceberg/blob/master/spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java#L229) here when trying to get the operation associated with the snapshot.
In this approach, planning fails altogether since for streaming it's required that there is a known chain of snapshot.
Although would appreciate feedback from folks more familiar with Spark @RussellSpitzer @aokolnychyi @singhpk234 @rajarshisarkar . Not sure if we can safely just skip the snapshot since a consumer of the stream technically is not consuming the original state of the table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 merged pull request #6480: Spark: Surface better error message during streaming planning when checkpoint snapshot not found
Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 merged PR #6480:
URL: https://github.com/apache/iceberg/pull/6480
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] cccs-jc commented on a diff in pull request #6480: Spark: Fail streaming planning when snapshot not found
Posted by "cccs-jc (via GitHub)" <gi...@apache.org>.
cccs-jc commented on code in PR #6480:
URL: https://github.com/apache/iceberg/pull/6480#discussion_r1166043802
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########
@@ -207,7 +207,14 @@ private List<FileScanTask> planFiles(StreamingOffset startOffset, StreamingOffse
currentOffset = new StreamingOffset(snapshotAfter.snapshotId(), 0L, false);
}
- if (!shouldProcess(table.snapshot(currentOffset.snapshotId()))) {
+ Snapshot snapshot = table.snapshot(currentOffset.snapshotId());
+
+ if (snapshot == null) {
+ throw new IllegalStateException(
+ String.format("Failed to find expected snapshot %d", snapshot.snapshotId()));
+ }
+
Review Comment:
agreed, if a snapshot is not found in the table, it's bad. It indicates the reader's checkpoint is too old. There is no nice way to recover from this, except clearing your checkpoints and restarting.
this happen to me because I was clearing my snapshots every hour. I realized then that this made no sense and that I should keep a week of snapshot. Readers have up to a week to fix any streaming job which went down.
So I'm all good with throwing an exception that points the user to the fact that the checkpointed snapshot ID is no longer available in the table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6480: Spark: Fail streaming planning when snapshot not found
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #6480:
URL: https://github.com/apache/iceberg/pull/6480#discussion_r1171678461
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########
@@ -212,7 +212,14 @@ private List<FileScanTask> planFiles(StreamingOffset startOffset, StreamingOffse
currentOffset = new StreamingOffset(snapshotAfter.snapshotId(), 0L, false);
}
- if (!shouldProcess(table.snapshot(currentOffset.snapshotId()))) {
+ Snapshot snapshot = table.snapshot(currentOffset.snapshotId());
+
+ if (snapshot == null) {
+ throw new IllegalStateException(
+ String.format("Failed to find expected snapshot %d", currentOffset.snapshotId()));
Review Comment:
Agreed, it'll be helpful to provide the context that expiration or some external mechanism somehow removed the snapshot, will update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6480: Spark: Fail streaming planning when snapshot not found
Posted by GitBox <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #6480:
URL: https://github.com/apache/iceberg/pull/6480#discussion_r1055070683
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########
@@ -207,7 +207,14 @@ private List<FileScanTask> planFiles(StreamingOffset startOffset, StreamingOffse
currentOffset = new StreamingOffset(snapshotAfter.snapshotId(), 0L, false);
}
- if (!shouldProcess(table.snapshot(currentOffset.snapshotId()))) {
+ Snapshot snapshot = table.snapshot(currentOffset.snapshotId());
+
+ if (snapshot == null) {
+ throw new IllegalStateException(
+ String.format("Failed to find expected snapshot %d", snapshot.snapshotId()));
+ }
+
Review Comment:
Will write a test for this case, if we determine if it's the right way to handle this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a diff in pull request #6480: Spark: Fail streaming planning when snapshot not found
Posted by GitBox <gi...@apache.org>.
nastra commented on code in PR #6480:
URL: https://github.com/apache/iceberg/pull/6480#discussion_r1055241225
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########
@@ -207,7 +207,14 @@ private List<FileScanTask> planFiles(StreamingOffset startOffset, StreamingOffse
currentOffset = new StreamingOffset(snapshotAfter.snapshotId(), 0L, false);
}
- if (!shouldProcess(table.snapshot(currentOffset.snapshotId()))) {
+ Snapshot snapshot = table.snapshot(currentOffset.snapshotId());
+
+ if (snapshot == null) {
+ throw new IllegalStateException(
+ String.format("Failed to find expected snapshot %d", snapshot.snapshotId()));
+ }
+
Review Comment:
I'm curious whether we could just add this to `shouldProcess`:
```
private boolean shouldProcess(Snapshot snapshot) {
+ if (null == snapshot) {
+ return false;
+ }
String op = snapshot.operation();
switch (op) {
case DataOperations.APPEND:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on pull request #6480: Spark: Surface better error message during streaming planning when checkpoint snapshot not found
Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on PR #6480:
URL: https://github.com/apache/iceberg/pull/6480#issuecomment-1515295300
Looks like the message is updated, thanks Amogh!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6480: Spark: Fail streaming planning when snapshot not found
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #6480:
URL: https://github.com/apache/iceberg/pull/6480#discussion_r1171557146
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########
@@ -212,7 +212,14 @@ private List<FileScanTask> planFiles(StreamingOffset startOffset, StreamingOffse
currentOffset = new StreamingOffset(snapshotAfter.snapshotId(), 0L, false);
}
- if (!shouldProcess(table.snapshot(currentOffset.snapshotId()))) {
+ Snapshot snapshot = table.snapshot(currentOffset.snapshotId());
+
+ if (snapshot == null) {
+ throw new IllegalStateException(
+ String.format("Failed to find expected snapshot %d", currentOffset.snapshotId()));
Review Comment:
I think we can probably improve this message. "Cannot load current offset at snapshot x, the snapshot was expired or removed"?
I'm trying to figure out how we can give the user a bit more information about what to do in this situation. Our currentOffset should always have a valid snapshot ID so I think it's probably ok to just say that that the snapshot was actually expired if we can't find it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #6480: Spark: Fail streaming planning when snapshot not found
Posted by GitBox <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #6480:
URL: https://github.com/apache/iceberg/pull/6480#discussion_r1055070683
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########
@@ -207,7 +207,14 @@ private List<FileScanTask> planFiles(StreamingOffset startOffset, StreamingOffse
currentOffset = new StreamingOffset(snapshotAfter.snapshotId(), 0L, false);
}
- if (!shouldProcess(table.snapshot(currentOffset.snapshotId()))) {
+ Snapshot snapshot = table.snapshot(currentOffset.snapshotId());
+
+ if (snapshot == null) {
+ throw new IllegalStateException(
+ String.format("Failed to find expected snapshot %d", snapshot.snapshotId()));
+ }
+
Review Comment:
Will write a test for this if we determine if it's the right way to handle this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] singhpk234 commented on a diff in pull request #6480: Spark: Fail streaming planning when snapshot not found
Posted by GitBox <gi...@apache.org>.
singhpk234 commented on code in PR #6480:
URL: https://github.com/apache/iceberg/pull/6480#discussion_r1059091796
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########
@@ -207,7 +207,14 @@ private List<FileScanTask> planFiles(StreamingOffset startOffset, StreamingOffse
currentOffset = new StreamingOffset(snapshotAfter.snapshotId(), 0L, false);
}
- if (!shouldProcess(table.snapshot(currentOffset.snapshotId()))) {
+ Snapshot snapshot = table.snapshot(currentOffset.snapshotId());
+
+ if (snapshot == null) {
+ throw new IllegalStateException(
+ String.format("Failed to find expected snapshot %d", snapshot.snapshotId()));
+ }
+
Review Comment:
as per my understanding returning `false` from shouldProcess will make it skip the expired snapshot, which might not be expected if the expired snapshot was of type `APPEND`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] amogh-jahagirdar commented on pull request #6480: Spark: Fail streaming planning when snapshot not found
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on PR #6480:
URL: https://github.com/apache/iceberg/pull/6480#issuecomment-1510761237
I've updated with a test, @aokolnychyi @RussellSpitzer @szehon-ho @singhpk234 @jackye1995 please take a look when you get a chance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org