You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by "e-mhui (via GitHub)" <gi...@apache.org> on 2023/07/28 07:31:17 UTC
[GitHub] [inlong] e-mhui opened a new pull request, #8607: [INLONG-8602][Sort] Fix StackOverflowError of Oracle CDC
e-mhui opened a new pull request, #8607:
URL: https://github.com/apache/inlong/pull/8607
### Prepare a Pull Request
[INLONG-8602][Sort] Fix StackOverflowError of Oracle CDC
- Fixes #8602
### Motivation
**DebeziumDeserializationSchema** has three methods:
- `void deserialize(SourceRecord record, Collector<T> out)`
- `default void deserialize(SourceRecord record, Collector<T> out, TableChanges.TableChange tableSchema)`
- `default void deserialize(SourceRecord record, Collector<T> out, Boolean isStreamingPhase)`
**RowDataDebeziumDeserializeSchema** is an implementation of **DebeziumDeserializationSchema**. When calling the `RowDataDebeziumDeserializeSchema.deserialize(SourceRecord record, Collector<RowData> out)` method, it invokes the `deserialize(record, out)` method, as shown in the code below:
```java
```
This results in a recursive call, causing StackOverflowError like #8602.
In fact, we need to call the `deserialize(SourceRecord record, Collector<RowData> out, TableChange tableSchema)` method. Previously, we could use `deserialize(record, out, null)`, but later the `deserialize(SourceRecord record, Collector<T> out, Boolean isStreamingPhase)` method was added, which conflicts with the `deserialize(SourceRecord record, Collector<RowData> out, TableChange tableSchema)` method.
<img width="1312" alt="image" src="https://github.com/apache/inlong/assets/111486498/cf8db1d4-f736-4b9c-aea6-68ce99b411da">
Therefore, we must override the `deserialize(SourceRecord record, Collector<RowData> out, TableChange tableSchema)` method.
### Modifications
Override the `RowDataDebeziumDeserializeSchema.deserialize(SourceRecord record, Collector<RowData> out, TableChange tableSchema)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on pull request #8607: [INLONG-8602][Sort] Fix StackOverflowError of Oracle CDC
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap commented on PR #8607:
URL: https://github.com/apache/inlong/pull/8607#issuecomment-1661372769
Great work ! same issue with Flink-cdc ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #8607: [INLONG-8602][Sort] Fix StackOverflowError of Oracle CDC
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap commented on code in PR #8607:
URL: https://github.com/apache/inlong/pull/8607#discussion_r1281452279
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/oracle-cdc/src/main/java/org/apache/inlong/sort/cdc/oracle/debezium/table/RowDataDebeziumDeserializeSchema.java:
##########
@@ -797,7 +797,32 @@ private Object getValueWithSchema(Object fieldValue, String schemaName) {
@Override
public void deserialize(SourceRecord record, Collector<RowData> out) throws Exception {
- deserialize(record, out);
+ Envelope.Operation op = Envelope.operationFor(record);
Review Comment:
There are same code in public void deserialize(SourceRecord record, Collector<RowData> out,TableChange tableSchema) , maybe reuse them is a better idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] dockerzhang merged pull request #8607: [INLONG-8602][Sort] Fix StackOverflowError of Oracle CDC
Posted by "dockerzhang (via GitHub)" <gi...@apache.org>.
dockerzhang merged PR #8607:
URL: https://github.com/apache/inlong/pull/8607
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on pull request #8607: [INLONG-8602][Sort] Fix StackOverflowError of Oracle CDC
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap commented on PR #8607:
URL: https://github.com/apache/inlong/pull/8607#issuecomment-1661442025
> > Great work ! same issue with Flink-cdc ?
>
> In the **RowDataDebeziumDeserializeSchema** of Flink CDC, there is only one method `void deserialize(SourceRecord record, Collector<T> out)`. However, in the **RowDataDebeziumDeserializeSchema** of INLONG, two new methods default `void deserialize(SourceRecord record, Collector<T> out, TableChanges.TableChange tableSchema)` and `default void deserialize(SourceRecord record, Collector<T> out, Boolean isStreamingPhase)` are added to implement the all database migration. Therefore, there will be no conflict issue in Flink CDC.
OK get it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] e-mhui commented on a diff in pull request #8607: [INLONG-8602][Sort] Fix StackOverflowError of Oracle CDC
Posted by "e-mhui (via GitHub)" <gi...@apache.org>.
e-mhui commented on code in PR #8607:
URL: https://github.com/apache/inlong/pull/8607#discussion_r1281566833
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/oracle-cdc/src/main/java/org/apache/inlong/sort/cdc/oracle/debezium/table/RowDataDebeziumDeserializeSchema.java:
##########
@@ -797,7 +797,32 @@ private Object getValueWithSchema(Object fieldValue, String schemaName) {
@Override
public void deserialize(SourceRecord record, Collector<RowData> out) throws Exception {
- deserialize(record, out);
+ Envelope.Operation op = Envelope.operationFor(record);
Review Comment:
Thanks, it has done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #8607: [INLONG-8602][Sort] Fix StackOverflowError of Oracle CDC
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap commented on code in PR #8607:
URL: https://github.com/apache/inlong/pull/8607#discussion_r1281452279
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/oracle-cdc/src/main/java/org/apache/inlong/sort/cdc/oracle/debezium/table/RowDataDebeziumDeserializeSchema.java:
##########
@@ -797,7 +797,32 @@ private Object getValueWithSchema(Object fieldValue, String schemaName) {
@Override
public void deserialize(SourceRecord record, Collector<RowData> out) throws Exception {
- deserialize(record, out);
+ Envelope.Operation op = Envelope.operationFor(record);
Review Comment:
There are some duplicate code in public void deserialize(SourceRecord record, Collector<RowData> out,TableChange tableSchema) , maybe reuse them is a better idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] e-mhui commented on pull request #8607: [INLONG-8602][Sort] Fix StackOverflowError of Oracle CDC
Posted by "e-mhui (via GitHub)" <gi...@apache.org>.
e-mhui commented on PR #8607:
URL: https://github.com/apache/inlong/pull/8607#issuecomment-1655460708
@dockerzhang Hi, pls help rerun the workflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] e-mhui commented on pull request #8607: [INLONG-8602][Sort] Fix StackOverflowError of Oracle CDC
Posted by "e-mhui (via GitHub)" <gi...@apache.org>.
e-mhui commented on PR #8607:
URL: https://github.com/apache/inlong/pull/8607#issuecomment-1661390251
> Great work ! same issue with Flink-cdc ?
In the **RowDataDebeziumDeserializeSchema** of Flink CDC, there is only one method `void deserialize(SourceRecord record, Collector<T> out)`. However, in the **RowDataDebeziumDeserializeSchema** of INLONG, two new methods default `void deserialize(SourceRecord record, Collector<T> out, TableChanges.TableChange tableSchema)` and `default void deserialize(SourceRecord record, Collector<T> out, Boolean isStreamingPhase)` are added to implement the all database migration. Therefore, there will be no conflict issue in Flink CDC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on pull request #8607: [INLONG-8602][Sort] Fix StackOverflowError of Oracle CDC
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap commented on PR #8607:
URL: https://github.com/apache/inlong/pull/8607#issuecomment-1661441446
> > Great work ! same issue with Flink-cdc ?
>
> In the **RowDataDebeziumDeserializeSchema** of Flink CDC, there is only one method `void deserialize(SourceRecord record, Collector<T> out)`. However, in the **RowDataDebeziumDeserializeSchema** of INLONG, two new methods default `void deserialize(SourceRecord record, Collector<T> out, TableChanges.TableChange tableSchema)` and `default void deserialize(SourceRecord record, Collector<T> out, Boolean isStreamingPhase)` are added to implement the all database migration. Therefore, there will be no conflict issue in Flink CDC.
Undertood !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org