You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by "e-mhui (via GitHub)" <gi...@apache.org> on 2023/04/25 07:41:20 UTC
[GitHub] [inlong] e-mhui opened a new pull request, #7917: [INLONG-7909][Sort] Fix Oracle CDC cannot capture ddl changes
e-mhui opened a new pull request, #7917:
URL: https://github.com/apache/inlong/pull/7917
### Prepare a Pull Request
[INLONG-7909][Sort] Fix Oracle CDC cannot capture ddl changesvv
- Fixes #7909
### Motivation
In debezium 1.6.4-final, the connector user, `SYS`, and `SYSTEM` users are prohibited from querying DDL operations, which prevents obtaining the latest schema. This causes errors due to inconsistent schema when inserting or updating data.
```
Caused by: io.debezium.connector.oracle.logminer.parser.DmlParserException: Failed to parse update DML: 'update "FLINKUSER"."TEST_04" set "Z" = 'ZZZ' where "ID" = '105' and "NAME" = '105' and "AGE" = '150' and "C" = '150' and "D" = '105' and "E" = '1111' and "F" = 'FFF' and "G" = 'GGG' and "I" = 'IIII' and "H" = 'HHH' and "Z" IS NULL;'
at io.debezium.connector.oracle.logminer.parser.LogMinerDmlParser.parseUpdate(LogMinerDmlParser.java:155) ~[debezium-connector-oracle-1.6.4.Final.jar:1.6.4.Final]
at io.debezium.connector.oracle.logminer.parser.LogMinerDmlParser.parse(LogMinerDmlParser.java:77) ~[debezium-connector-oracle-1.6.4.Final.jar:1.6.4.Final]
at io.debezium.connector.oracle.logminer.LogMinerQueryResultProcessor.parse(LogMinerQueryResultProcessor.java:397) ~[debezium-connector-oracle-1.6.4.Final.jar:1.6.4.Final]
... 14 more
Caused by: io.debezium.DebeziumException: No column 'Z' found in table 'XE.FLINKUSER.TEST_04'
```
This has been fixed in debezium 1.7.0-final. For details, please refer to: https://github.com/debezium/debezium/pull/2510.
### Modifications
We should remove the restrictions on the connector user but keep the restrictions on `SYS` and `SYSTEM`.
```java
/**
* Builds a common SQL fragment used to obtain DDL operations via LogMiner.
*
* @return predicate that can be used to obtain DDL operations via LogMiner
*/
private static String buildDdlPredicate() {
final StringBuilder predicate = new StringBuilder(256);
predicate.append("(OPERATION_CODE = 5 ");
predicate.append("AND USERNAME NOT IN ('SYS','SYSTEM') ");
predicate.append("AND INFO NOT LIKE 'INTERNAL DDL%' ");
predicate.append("AND (TABLE_NAME IS NULL OR TABLE_NAME NOT LIKE 'ORA_TEMP_%'))");
return predicate.toString();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] dockerzhang merged pull request #7917: [INLONG-7909][Sort] Fix Oracle CDC cannot capture ddl changes
Posted by "dockerzhang (via GitHub)" <gi...@apache.org>.
dockerzhang merged PR #7917:
URL: https://github.com/apache/inlong/pull/7917
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #7917: [INLONG-7909][Sort] Fix Oracle CDC cannot capture ddl changes
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap commented on code in PR #7917:
URL: https://github.com/apache/inlong/pull/7917#discussion_r1176195009
##########
inlong-sort/sort-connectors/oracle-cdc/src/main/java/io/debezium/connector/oracle/logminer/LogMinerQueryBuilder.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package io.debezium.connector.oracle.logminer;
Review Comment:
package name should be start with org.apache.inlong?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] e-mhui commented on a diff in pull request #7917: [INLONG-7909][Sort] Fix Oracle CDC cannot capture ddl changes
Posted by "e-mhui (via GitHub)" <gi...@apache.org>.
e-mhui commented on code in PR #7917:
URL: https://github.com/apache/inlong/pull/7917#discussion_r1176212420
##########
inlong-sort/sort-connectors/oracle-cdc/src/main/java/io/debezium/connector/oracle/logminer/LogMinerQueryBuilder.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package io.debezium.connector.oracle.logminer;
Review Comment:
The purpose is to modify the source code of debezium. If the package name is changed to start with 'inlong', which to be copied a large amount of irrelevant debezium code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] dockerzhang commented on pull request #7917: [INLONG-7909][Sort] Fix Oracle CDC cannot capture ddl changes
Posted by "dockerzhang (via GitHub)" <gi...@apache.org>.
dockerzhang commented on PR #7917:
URL: https://github.com/apache/inlong/pull/7917#issuecomment-1521502180
Can we publish to the upstream of the debezium and then depend on it, instead of copying the code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #7917: [INLONG-7909][Sort] Fix Oracle CDC cannot capture ddl changes
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap commented on code in PR #7917:
URL: https://github.com/apache/inlong/pull/7917#discussion_r1176196037
##########
inlong-sort/sort-connectors/oracle-cdc/src/main/java/io/debezium/connector/oracle/logminer/LogMinerQueryBuilder.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package io.debezium.connector.oracle.logminer;
Review Comment:
package name should start with org.apache.inlong ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] e-mhui commented on pull request #7917: [INLONG-7909][Sort] Fix Oracle CDC cannot capture ddl changes
Posted by "e-mhui (via GitHub)" <gi...@apache.org>.
e-mhui commented on PR #7917:
URL: https://github.com/apache/inlong/pull/7917#issuecomment-1521521708
> Can we publish to the upstream of the debezium and then depend on it, instead of copying the code?
The version of Debezium 1.6.4-Final that we depend on has been discontinued, and we cannot push code to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org