You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "yuxiqian (via GitHub)" <gi...@apache.org> on 2024/04/11 08:29:43 UTC
[PR] [FLINK-35079] Fallback to timestamp startup mode when resume token has expired [flink-cdc]
yuxiqian opened a new pull request, #3221:
URL: https://github.com/apache/flink-cdc/pull/3221
Closes [FLINK-35079](https://issues.apache.org/jira/browse/FLINK-35079).
When MongoDB CDC connector tries to create cursor with an expired resuming token during stream task fetching stage, it will crash with a fatal exception: `error due to Command failed with error 280 (ChangeStreamFatalError): cannot resume stream; the resume token was not found.`
This PR added fallback logic to create cursor with timestamp, which only runs when:
* Mongo CDC is in StreamTaskFetch stage
* a `ChangeStreamFatalError` (280) is raised
* Current `ChangeStreamOffset` has a valid timestamp field
Also tweaked @loserwang1024 's test case to cover resume token expiration scenario better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35079] Fallback to timestamp startup mode when resume token has expired [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #3221:
URL: https://github.com/apache/flink-cdc/pull/3221#discussion_r1560784586
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mongodb-cdc/src/main/java/org/apache/flink/cdc/connectors/mongodb/source/reader/fetch/MongoDBStreamFetchTask.java:
##########
@@ -108,7 +110,32 @@ public void execute(Context context) throws Exception {
this.taskRunning = true;
try {
while (taskRunning) {
- Optional<BsonDocument> next = Optional.ofNullable(changeStreamCursor.tryNext());
+ Optional<BsonDocument> next;
+ try {
+ next = Optional.ofNullable(changeStreamCursor.tryNext());
+ } catch (MongoQueryException e) {
+ if (e.getErrorCode() == CHANGE_STREAM_FATAL_ERROR) {
+ ChangeStreamOffset offset =
+ new ChangeStreamOffset(streamSplit.getStartingOffset().getOffset());
Review Comment:
Yes, it would be much cleaner to move that checking logic to `openChangeStreamCursor` and implement the fallback logic along with existing routes.
CMIIW but seems `MongoCommandException` will only be thrown [when getNext or tryGetNext is called](https://github.com/mongodb/mongo/blob/a71feef352a1f08e3f9dbc2b840691e75775a370/src/mongo/db/pipeline/document_source_change_stream_ensure_resume_token_present.cpp#L133C31-L133C83), and we can't intentionally trigger an `MongoCommandException` by calling other methods like `hasNext` earlier.
@Jiabao-Sun Any suggestions about this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35079] Fallback to timestamp startup mode when resume token has expired [flink-cdc]
Posted by "Jiabao-Sun (via GitHub)" <gi...@apache.org>.
Jiabao-Sun commented on code in PR #3221:
URL: https://github.com/apache/flink-cdc/pull/3221#discussion_r1560798123
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mongodb-cdc/src/main/java/org/apache/flink/cdc/connectors/mongodb/source/reader/fetch/MongoDBStreamFetchTask.java:
##########
@@ -108,7 +110,32 @@ public void execute(Context context) throws Exception {
this.taskRunning = true;
try {
while (taskRunning) {
- Optional<BsonDocument> next = Optional.ofNullable(changeStreamCursor.tryNext());
+ Optional<BsonDocument> next;
+ try {
+ next = Optional.ofNullable(changeStreamCursor.tryNext());
+ } catch (MongoQueryException e) {
+ if (e.getErrorCode() == CHANGE_STREAM_FATAL_ERROR) {
+ ChangeStreamOffset offset =
+ new ChangeStreamOffset(streamSplit.getStartingOffset().getOffset());
Review Comment:
Hi @yuxiqian, the `MongoCommandException` seems can be thrown by openChangeStreamCursor.
```
at com.mongodb.client.internal.ChangeStreamIterableImpl$1.cursor(ChangeStreamIterableImpl.java:121)
at com.ververica.cdc.connectors.mongodb.source.reader.fetch.MongoDBStreamFetchTask.openChangeStreamCursor(MongoDBStreamFetchTask.java:237)
```

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35079] Fallback to timestamp startup mode when resume token has expired [flink-cdc]
Posted by "Jiabao-Sun (via GitHub)" <gi...@apache.org>.
Jiabao-Sun commented on code in PR #3221:
URL: https://github.com/apache/flink-cdc/pull/3221#discussion_r1560833766
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mongodb-cdc/src/main/java/org/apache/flink/cdc/connectors/mongodb/source/reader/fetch/MongoDBStreamFetchTask.java:
##########
@@ -108,7 +110,19 @@ public void execute(Context context) throws Exception {
this.taskRunning = true;
try {
while (taskRunning) {
- Optional<BsonDocument> next = Optional.ofNullable(changeStreamCursor.tryNext());
+ Optional<BsonDocument> next;
+ try {
+ next = Optional.ofNullable(changeStreamCursor.tryNext());
+ } catch (MongoCommandException e) {
+ if (MongoUtils.checkIfResumeTokenExpires(e)) {
+ resumeTokenExpired = true;
Review Comment:
Do we need to reset its value?
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mongodb-cdc/src/main/java/org/apache/flink/cdc/connectors/mongodb/source/reader/fetch/MongoDBStreamFetchTask.java:
##########
@@ -236,7 +250,7 @@ private MongoChangeStreamCursor<BsonDocument> openChangeStreamCursor(
BsonDocument resumeToken = offset.getResumeToken();
BsonTimestamp timestamp = offset.getTimestamp();
- if (resumeToken != null) {
+ if (resumeToken != null && !resumeTokenExpired) {
Review Comment:
It should be noted that `startAtOperationTime` is supported starting from MongoDB version 4.0.
The prerequisite for executing the following code block is that the `resumeToken` does not exist, which occurs in two scenarios:
1. a specified timestamp is used as the starting offset.
2. prior to version 4.0.7, when a collection has not been updated for a long time and the `postResumeToken` cannot be obtained, the current timestamp is used as the starting offset.
In the `else` branch of the code below, if we cannot start the Change Stream using `startAtOperationTime ` , we fallback to starting the Change Stream from the latest offset.
Before MongoDB 4.0, we may lose data during the snapshot changes, but the likelihood of losing data during the snapshot is relatively low: if a collection is highly likely to change, we can obtain the `resumeToken` at its starting position; if a collection has not changed for a long time and we cannot obtain the `resumeToken` and `postResumeToken`, the likelihood of data loss during the snapshot is very low.
However, when we encounter an invalid `resumeToken` during runtime or when recovering from a checkpoint, there is a high possibility of data loss. I have reconsidered this issue, and if we cannot recover the change stream from the `resumeToken`, it is highly likely that we cannot recover it through `startAtOperationTime` either, as they correspond to the same position in the `oplog`.
In `mongo-kafka`, there is an explicit configuration called "tolerant-errors" provided to handle interruptions in the change stream. But in some scenarios, consistency requirements may outweigh availability, we should throw an exception to let user re-run the task from beginning.
```java
else {
if (supportsStartAtOperationTime) {
LOG.info("Open the change stream at the timestamp: {}", timestamp);
changeStreamIterable.startAtOperationTime(timestamp);
} else {
LOG.warn("Open the change stream of the latest offset");
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35079] Fallback to timestamp startup mode when resume token has expired [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #3221:
URL: https://github.com/apache/flink-cdc/pull/3221#discussion_r1566596993
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mongodb-cdc/src/main/java/org/apache/flink/cdc/connectors/mongodb/source/reader/fetch/MongoDBStreamFetchTask.java:
##########
@@ -283,6 +301,9 @@ private MongoChangeStreamCursor<BsonDocument> openChangeStreamCursor(
e.getErrorMessage(),
e.getErrorCode());
throw new FlinkRuntimeException("Unauthorized $changeStream operation", e);
+ } else if (timestamp != null && MongoUtils.checkIfResumeTokenExpires(e)) {
+ ((ChangeStreamOffset) streamSplit.getStartingOffset()).clearResumeToken();
+ return openChangeStreamCursor(changeStreamDescriptor);
Review Comment:
Yes, it would be cleaner to add a switch parameter instead of keeping global variable / status elsewhere. Pushed another commit to solve this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35079] Fallback to timestamp startup mode when resume token has expired [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #3221:
URL: https://github.com/apache/flink-cdc/pull/3221#discussion_r1560819276
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mongodb-cdc/src/main/java/org/apache/flink/cdc/connectors/mongodb/source/reader/fetch/MongoDBStreamFetchTask.java:
##########
@@ -108,7 +110,32 @@ public void execute(Context context) throws Exception {
this.taskRunning = true;
try {
while (taskRunning) {
- Optional<BsonDocument> next = Optional.ofNullable(changeStreamCursor.tryNext());
+ Optional<BsonDocument> next;
+ try {
+ next = Optional.ofNullable(changeStreamCursor.tryNext());
+ } catch (MongoQueryException e) {
+ if (e.getErrorCode() == CHANGE_STREAM_FATAL_ERROR) {
+ ChangeStreamOffset offset =
+ new ChangeStreamOffset(streamSplit.getStartingOffset().getOffset());
Review Comment:
@Jiabao-Sun You're right, I've added resume token expiration checks in creating and iterating process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35079] Fallback to timestamp startup mode when resume token has expired [flink-cdc]
Posted by "Jiabao-Sun (via GitHub)" <gi...@apache.org>.
Jiabao-Sun commented on code in PR #3221:
URL: https://github.com/apache/flink-cdc/pull/3221#discussion_r1560751221
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mongodb-cdc/src/main/java/org/apache/flink/cdc/connectors/mongodb/source/reader/fetch/MongoDBStreamFetchTask.java:
##########
@@ -108,7 +110,32 @@ public void execute(Context context) throws Exception {
this.taskRunning = true;
try {
while (taskRunning) {
- Optional<BsonDocument> next = Optional.ofNullable(changeStreamCursor.tryNext());
+ Optional<BsonDocument> next;
+ try {
+ next = Optional.ofNullable(changeStreamCursor.tryNext());
+ } catch (MongoQueryException e) {
+ if (e.getErrorCode() == CHANGE_STREAM_FATAL_ERROR) {
+ ChangeStreamOffset offset =
+ new ChangeStreamOffset(streamSplit.getStartingOffset().getOffset());
Review Comment:
I think we can refer to the logic in mongo-kafka to reopen the changestream.
https://github.com/mongodb/mongo-kafka/blob/2ffd620441b68bec19227dbfa641375d37148e39/src/main/java/com/mongodb/kafka/connect/source/StartedMongoSourceTask.java#L102
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35079] Fallback to timestamp startup mode when resume token has expired [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on PR #3221:
URL: https://github.com/apache/flink-cdc/pull/3221#issuecomment-2051530489
Resume token could be refreshed by processing any data record or heartbeat event. When a resume token carried by a `ChangeStreamOffset` is found to be expired, resume token field is removed from that offset, and will fallback to timestamp mode automatically only after another event updates that offset. Though it's clearer than setting flag bits but it might not be the ideal solution since it changes given data structures. Comments are welcomed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35079] Fallback to timestamp startup mode when resume token has expired [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #3221:
URL: https://github.com/apache/flink-cdc/pull/3221#discussion_r1560712067
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mongodb-cdc/src/main/java/org/apache/flink/cdc/connectors/mongodb/source/reader/fetch/MongoDBStreamFetchTask.java:
##########
@@ -108,7 +110,32 @@ public void execute(Context context) throws Exception {
this.taskRunning = true;
try {
while (taskRunning) {
- Optional<BsonDocument> next = Optional.ofNullable(changeStreamCursor.tryNext());
+ Optional<BsonDocument> next = Optional.empty();
+ try {
+ next = Optional.ofNullable(changeStreamCursor.tryNext());
+ } catch (MongoQueryException e) {
+ if (e.getErrorCode() == CHANGE_STREAM_FATAL_ERROR) {
+ ChangeStreamOffset offset =
+ new ChangeStreamOffset(streamSplit.getStartingOffset().getOffset());
+ if (offset.getTimestamp() != null) {
+ LOG.warn(
+ "Resume token {} has expired, fallback to timestamp startup mode.",
Review Comment:
Addressed that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35079] Fallback to timestamp startup mode when resume token has expired [flink-cdc]
Posted by "Jiabao-Sun (via GitHub)" <gi...@apache.org>.
Jiabao-Sun commented on code in PR #3221:
URL: https://github.com/apache/flink-cdc/pull/3221#discussion_r1565666103
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mongodb-cdc/src/main/java/org/apache/flink/cdc/connectors/mongodb/source/reader/fetch/MongoDBStreamFetchTask.java:
##########
@@ -283,6 +301,9 @@ private MongoChangeStreamCursor<BsonDocument> openChangeStreamCursor(
e.getErrorMessage(),
e.getErrorCode());
throw new FlinkRuntimeException("Unauthorized $changeStream operation", e);
+ } else if (timestamp != null && MongoUtils.checkIfResumeTokenExpires(e)) {
+ ((ChangeStreamOffset) streamSplit.getStartingOffset()).clearResumeToken();
+ return openChangeStreamCursor(changeStreamDescriptor);
Review Comment:
I prefer to add a parameter to `openChangeStreamCursor` to indicate forcing the use of timestamp mode instead of clearing the resumeToken.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35079] Fallback to timestamp startup mode when resume token has expired [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #3221:
URL: https://github.com/apache/flink-cdc/pull/3221#discussion_r1560784586
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mongodb-cdc/src/main/java/org/apache/flink/cdc/connectors/mongodb/source/reader/fetch/MongoDBStreamFetchTask.java:
##########
@@ -108,7 +110,32 @@ public void execute(Context context) throws Exception {
this.taskRunning = true;
try {
while (taskRunning) {
- Optional<BsonDocument> next = Optional.ofNullable(changeStreamCursor.tryNext());
+ Optional<BsonDocument> next;
+ try {
+ next = Optional.ofNullable(changeStreamCursor.tryNext());
+ } catch (MongoQueryException e) {
+ if (e.getErrorCode() == CHANGE_STREAM_FATAL_ERROR) {
+ ChangeStreamOffset offset =
+ new ChangeStreamOffset(streamSplit.getStartingOffset().getOffset());
Review Comment:
Yes, it would be much cleaner to move that checking logic to `openChangeStreamCursor` and implement the fallback logic along with existing routes.
CMIIW but seems `MongoCommandException` will only be thrown [when getNext or tryGetNext is called](https://github.com/mongodb/mongo/blob/a71feef352a1f08e3f9dbc2b840691e75775a370/src/mongo/db/pipeline/document_source_change_stream_ensure_resume_token_present.cpp#L133C31-L133C83), and we can't intentionally trigger an `MongoCommandException` by calling other methods like `hasNext` earlier.
@Jiabao-Sun Any suggestions about this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35079] Fallback to timestamp startup mode when resume token has expired [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #3221:
URL: https://github.com/apache/flink-cdc/pull/3221#discussion_r1562397462
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mongodb-cdc/src/main/java/org/apache/flink/cdc/connectors/mongodb/source/reader/fetch/MongoDBStreamFetchTask.java:
##########
@@ -283,6 +301,9 @@ private MongoChangeStreamCursor<BsonDocument> openChangeStreamCursor(
e.getErrorMessage(),
e.getErrorCode());
throw new FlinkRuntimeException("Unauthorized $changeStream operation", e);
+ } else if (timestamp != null && MongoUtils.checkIfResumeTokenExpires(e)) {
+ ((ChangeStreamOffset) streamSplit.getStartingOffset()).clearResumeToken();
+ return openChangeStreamCursor(changeStreamDescriptor);
Review Comment:
Recursively calling `openChangeStreamCursor` is safe here, since resume token is cleared here, and `checkResumeTokenExpires` would never be true again since no resume token is provided.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35079] Fallback to timestamp startup mode when resume token has expired [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #3221:
URL: https://github.com/apache/flink-cdc/pull/3221#discussion_r1566596993
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mongodb-cdc/src/main/java/org/apache/flink/cdc/connectors/mongodb/source/reader/fetch/MongoDBStreamFetchTask.java:
##########
@@ -283,6 +301,9 @@ private MongoChangeStreamCursor<BsonDocument> openChangeStreamCursor(
e.getErrorMessage(),
e.getErrorCode());
throw new FlinkRuntimeException("Unauthorized $changeStream operation", e);
+ } else if (timestamp != null && MongoUtils.checkIfResumeTokenExpires(e)) {
+ ((ChangeStreamOffset) streamSplit.getStartingOffset()).clearResumeToken();
+ return openChangeStreamCursor(changeStreamDescriptor);
Review Comment:
Yes, that would be cleaner to add a switch parameter instead of keeping global variable / status elsewhere. Pushed another commit to solve that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35079] Fallback to timestamp startup mode when resume token has expired [flink-cdc]
Posted by "Jiabao-Sun (via GitHub)" <gi...@apache.org>.
Jiabao-Sun merged PR #3221:
URL: https://github.com/apache/flink-cdc/pull/3221
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35079] Fallback to timestamp startup mode when resume token has expired [flink-cdc]
Posted by "Jiabao-Sun (via GitHub)" <gi...@apache.org>.
Jiabao-Sun commented on code in PR #3221:
URL: https://github.com/apache/flink-cdc/pull/3221#discussion_r1560736878
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mongodb-cdc/src/main/java/org/apache/flink/cdc/connectors/mongodb/source/reader/fetch/MongoDBStreamFetchTask.java:
##########
@@ -108,7 +110,32 @@ public void execute(Context context) throws Exception {
this.taskRunning = true;
try {
while (taskRunning) {
- Optional<BsonDocument> next = Optional.ofNullable(changeStreamCursor.tryNext());
+ Optional<BsonDocument> next;
+ try {
+ next = Optional.ofNullable(changeStreamCursor.tryNext());
+ } catch (MongoQueryException e) {
+ if (e.getErrorCode() == CHANGE_STREAM_FATAL_ERROR) {
+ ChangeStreamOffset offset =
+ new ChangeStreamOffset(streamSplit.getStartingOffset().getOffset());
Review Comment:
Thanks @yuxiqian for this fix.
Why don't we catch this exception when we are in `openChangeStreamCursor`?
Is there any consideration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35079] Fallback to timestamp startup mode when resume token has expired [flink-cdc]
Posted by "loserwang1024 (via GitHub)" <gi...@apache.org>.
loserwang1024 commented on code in PR #3221:
URL: https://github.com/apache/flink-cdc/pull/3221#discussion_r1560701972
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mongodb-cdc/src/main/java/org/apache/flink/cdc/connectors/mongodb/source/reader/fetch/MongoDBStreamFetchTask.java:
##########
@@ -108,7 +110,32 @@ public void execute(Context context) throws Exception {
this.taskRunning = true;
try {
while (taskRunning) {
- Optional<BsonDocument> next = Optional.ofNullable(changeStreamCursor.tryNext());
+ Optional<BsonDocument> next = Optional.empty();
+ try {
+ next = Optional.ofNullable(changeStreamCursor.tryNext());
+ } catch (MongoQueryException e) {
+ if (e.getErrorCode() == CHANGE_STREAM_FATAL_ERROR) {
+ ChangeStreamOffset offset =
+ new ChangeStreamOffset(streamSplit.getStartingOffset().getOffset());
+ if (offset.getTimestamp() != null) {
+ LOG.warn(
+ "Resume token {} has expired, fallback to timestamp startup mode.",
Review Comment:
Still initial startup mode or latest mode, maybe "Resume token {} has expired, fallback to restart based on timestamp"?
I'm not a mongodb expert. I wonder whether some data will loss or duplicated when fallback from resume token to timestamp? @Jiabao-Sun , CC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35079] Fallback to timestamp startup mode when resume token has expired [flink-cdc]
Posted by "Jiabao-Sun (via GitHub)" <gi...@apache.org>.
Jiabao-Sun commented on code in PR #3221:
URL: https://github.com/apache/flink-cdc/pull/3221#discussion_r1560746786
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mongodb-cdc/src/main/java/org/apache/flink/cdc/connectors/mongodb/source/reader/fetch/MongoDBStreamFetchTask.java:
##########
@@ -108,7 +110,32 @@ public void execute(Context context) throws Exception {
this.taskRunning = true;
try {
while (taskRunning) {
- Optional<BsonDocument> next = Optional.ofNullable(changeStreamCursor.tryNext());
+ Optional<BsonDocument> next;
+ try {
+ next = Optional.ofNullable(changeStreamCursor.tryNext());
+ } catch (MongoQueryException e) {
+ if (e.getErrorCode() == CHANGE_STREAM_FATAL_ERROR) {
+ ChangeStreamOffset offset =
+ new ChangeStreamOffset(streamSplit.getStartingOffset().getOffset());
Review Comment:
I checked the following stack trace in #1879, and this exception occurs when opening the cursor.
```
at com.mongodb.internal.operation.OperationHelper.withReadConnectionSource(OperationHelper.java:583)
at com.mongodb.internal.operation.CommandOperationHelper.executeCommand(CommandOperationHelper.java:189)
at com.mongodb.internal.operation.AggregateOperationImpl.execute(AggregateOperationImpl.java:195)
at com.mongodb.internal.operation.ChangeStreamOperation$1.call(ChangeStreamOperation.java:323)
at com.mongodb.internal.operation.ChangeStreamOperation$1.call(ChangeStreamOperation.java:319)
at com.mongodb.internal.operation.OperationHelper.withReadConnectionSource(OperationHelper.java:583)
at com.mongodb.internal.operation.ChangeStreamOperation.execute(ChangeStreamOperation.java:319)
at com.mongodb.internal.operation.ChangeStreamOperation.execute(ChangeStreamOperation.java:58)
at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:184)
at com.mongodb.client.internal.ChangeStreamIterableImpl.execute(ChangeStreamIterableImpl.java:204)
at com.mongodb.client.internal.ChangeStreamIterableImpl.access$000(ChangeStreamIterableImpl.java:53)
at com.mongodb.client.internal.ChangeStreamIterableImpl$1.cursor(ChangeStreamIterableImpl.java:129)
at com.mongodb.client.internal.ChangeStreamIterableImpl$1.cursor(ChangeStreamIterableImpl.java:121)
at com.ververica.cdc.connectors.mongodb.source.reader.fetch.MongoDBStreamFetchTask.openChangeStreamCursor(MongoDBStreamFetchTask.java:237)
... 8 more
when use mongodb cdc (sql) transfer the mongodb table to flink table store table,if enable the scan.incremental.snapshot, the job will appear the error: " the resume token was not found" immediately.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org