You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@inlong.apache.org by "ComplexA (via GitHub)" <gi...@apache.org> on 2024/07/25 07:49:24 UTC
[I] [Bug]During the execution of the InLong Sort-Oracle-CDC task, the Extract node generates a large number of archive logs, and the Load node fails to capture incremental change data. [inlong]
ComplexA opened a new issue, #10719:
URL: https://github.com/apache/inlong/issues/10719
### What happened
While using the InLong Sort component for an Oracle-CDC task, the Extract node, running Oracle 11g, generated a significant volume of archive logs during task execution. The task only performed a one-time synchronization of full data upon startup. Although FlinkSQL was configured with incremental snapshot parameters, the Load node did not reflect corresponding data changes during data variations at the Extract node.
### What you expected to happen
I think that during task execution, Oracle database generated a large number of unknown archive logs, causing the task to fail in capturing the correct archive logs corresponding to data changes.
### How to reproduce
Configure the Extract node for Oracle 11g and the Load node for Oracle 19c, and initiate incremental snapshot synchronization.
### Environment
Windows 11 configuration with CentOS 7 virtual machine
### InLong version
1.12.0
### InLong Component
InLong Sort
### Are you willing to submit PR?
- [X] Yes, I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@inlong.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]During the execution of the InLong Sort-Oracle-CDC task, the Extract node generates a large number of archive logs, and the Load node fails to capture incremental change data. [inlong]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10719:
URL: https://github.com/apache/inlong/issues/10719#issuecomment-2249682449
Hello @ComplexA, thank you for opening your first issue in InLong ๐งก We will respond as soon as possible โณ
If this is a bug report, please provide screenshots or error logs for us to reproduce your issue, so we can do our best to fix it.
If you have any questions in the meantime, you can also ask us on the [InLong Discussions](https://github.com/apache/inlong/discussions) ๐
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]During the execution of the InLong Sort-Oracle-CDC task, the Extract node generates a large number of archive logs, and the Load node fails to capture incremental change data. [inlong]
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong commented on issue #10719:
URL: https://github.com/apache/inlong/issues/10719#issuecomment-2252040394
@ComplexA Hello. Could you add some taskmanager log info? Maybe, you can try add table option 'debezium.log.mining.strategy' = 'online_catalog' for extract node.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]During the execution of the InLong Sort-Oracle-CDC task, the Extract node generates a large number of archive logs, and the Load node fails to capture incremental change data. [inlong]
Posted by "e-mhui (via GitHub)" <gi...@apache.org>.
e-mhui commented on issue #10719:
URL: https://github.com/apache/inlong/issues/10719#issuecomment-2249841987
@ComplexA Maybe you haven't enable log archiving for Oracle. Refer [setup-oracle](https://inlong.apache.org/docs/data_node/extract_node/oracle-cdc#setup-oracle) to enable it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]During the execution of the InLong Sort-Oracle-CDC task, the Extract node generates a large number of archive logs, and the Load node fails to capture incremental change data. [inlong]
Posted by "vernedeng (via GitHub)" <gi...@apache.org>.
vernedeng commented on issue #10719:
URL: https://github.com/apache/inlong/issues/10719#issuecomment-2249797689
@gong @e-mhui
Hi, please take a look at this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]During the execution of the InLong Sort-Oracle-CDC task, the Extract node generates a large number of archive logs, and the Load node fails to capture incremental change data. [inlong]
Posted by "ComplexA (via GitHub)" <gi...@apache.org>.
ComplexA commented on issue #10719:
URL: https://github.com/apache/inlong/issues/10719#issuecomment-2250116432
> @ComplexA Would you please offer more information such as your connector options, and stream sink configuration
connector information :
```
CREATE TABLE `oracle11_extract_table` (
`ID` INT,
`NAME` STRING,
`CREATED_TIME` TIMESTAMP,
PRIMARY KEY (`ID`) NOT ENFORCED
) WITH (
'connector' = 'oracle-cdc-inlong',
'hostname' = '192.168.41.128',
'port' = '21521',
'username' = 'logminer_user',
'password' = 'oracle',
'database-name' = 'XE',
'schema-name' = 'logminer_user',
'table-name' = 'logminer_user.oraclecdc',
'debezium.snapshot.mode' = 'initial',
'scan.incremental.snapshot.enabled' = 'true'
);
CREATE TABLE `oracle19_load_table` (
`ID` INT,
`NAME` STRING,
`CREATED_TIME` TIMESTAMP,
PRIMARY KEY (`ID`) NOT ENFORCED
) WITH (
'connector' = 'jdbc-inlong',
'url' = 'jdbc:oracle:thin:@192.168.41.130:11521:ORCL',
'username' = 'flinkuser',
'password' = 'flinkpw',
'table-name' = 'flinkuser.ORACLE19CDC'
);
INSERT INTO `oracle11_extract_table`
SELECT `ID`, `NAME`, `CREATED_TIME` FROM `oracle19_load_table`;
```
flink-conf.yaml
```
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
#==============================================================================
# Common
#==============================================================================
# The external address of the host on which the JobManager runs and can be
# reached by the TaskManagers and any clients which want to connect. This setting
# is only used in Standalone mode and may be overwritten on the JobManager side
# by specifying the --host <hostname> parameter of the bin/jobmanager.sh executable.
# In high availability mode, if you use the bin/start-cluster.sh script and setup
# the conf/masters file, this will be taken care of automatically. Yarn/Mesos
# automatically configure the host name based on the hostname of the node where the
# JobManager runs.
jobmanager.rpc.address: localhost
# The RPC port where the JobManager is reachable.
jobmanager.rpc.port: 6123
# The total process memory size for the JobManager.
#
# Note this accounts for all memory usage within the JobManager process, including JVM metaspace and other overhead.
jobmanager.memory.process.size: 1600m
# The total process memory size for the TaskManager.
#
# Note this accounts for all memory usage within the TaskManager process, including JVM metaspace and other overhead.
taskmanager.memory.process.size: 1728m
# To exclude JVM metaspace and overhead, please, use total Flink memory size instead of 'taskmanager.memory.process.size'.
# It is not recommended to set both 'taskmanager.memory.process.size' and Flink memory.
#
# taskmanager.memory.flink.size: 1280m
# The number of task slots that each TaskManager offers. Each slot runs one parallel pipeline.
taskmanager.numberOfTaskSlots: 1
# The parallelism used for programs that did not specify and other parallelism.
parallelism.default: 1
# The default file system scheme and authority.
#
# By default file paths without scheme are interpreted relative to the local
# root file system 'file:///'. Use this to override the default and interpret
# relative paths relative to a different file system,
# for example 'hdfs://mynamenode:12345'
#
# fs.default-scheme
#==============================================================================
# High Availability
#==============================================================================
# The high-availability mode. Possible options are 'NONE' or 'zookeeper'.
#
# high-availability: zookeeper
# The path where metadata for master recovery is persisted. While ZooKeeper stores
# the small ground truth for checkpoint and leader election, this location stores
# the larger objects, like persisted dataflow graphs.
#
# Must be a durable file system that is accessible from all nodes
# (like HDFS, S3, Ceph, nfs, ...)
#
# high-availability.storageDir: hdfs:///flink/ha/
# The list of ZooKeeper quorum peers that coordinate the high-availability
# setup. This must be a list of the form:
# "host1:clientPort,host2:clientPort,..." (default clientPort: 2181)
#
# high-availability.zookeeper.quorum: localhost:2181
# ACL options are based on https://zookeeper.apache.org/doc/r3.1.2/zookeeperProgrammers.html#sc_BuiltinACLSchemes
# It can be either "creator" (ZOO_CREATE_ALL_ACL) or "open" (ZOO_OPEN_ACL_UNSAFE)
# The default value is "open" and it can be changed to "creator" if ZK security is enabled
#
# high-availability.zookeeper.client.acl: open
#==============================================================================
# Fault tolerance and checkpointing
#==============================================================================
# The backend that will be used to store operator state checkpoints if
# checkpointing is enabled.
#
# Supported backends are 'jobmanager', 'filesystem', 'rocksdb', or the
# <class-name-of-factory>.
#
# state.backend: filesystem
# Directory for checkpoints filesystem, when using any of the default bundled
# state backends.
#
# state.checkpoints.dir: hdfs://namenode-host:port/flink-checkpoints
# Default target directory for savepoints, optional.
#
# state.savepoints.dir: hdfs://namenode-host:port/flink-savepoints
# Flag to enable/disable incremental checkpoints for backends that
# support incremental checkpoints (like the RocksDB state backend).
#
# state.backend.incremental: false
# The failover strategy, i.e., how the job computation recovers from task failures.
# Only restart tasks that may have been affected by the task failure, which typically includes
# downstream tasks and potentially upstream tasks if their produced data is no longer available for consumption.
jobmanager.execution.failover-strategy: region
#==============================================================================
# Rest & web frontend
#==============================================================================
# The port to which the REST client connects to. If rest.bind-port has
# not been specified, then the server will bind to this port as well.
#
#rest.port: 8081
# The address to which the REST client will connect to
#
#rest.address: 0.0.0.0
# Port range for the REST and web server to bind to.
#
#rest.bind-port: 8080-8090
# The address that the REST & web server binds to
#
#rest.bind-address: 0.0.0.0
# Flag to specify whether job submission is enabled from the web-based
# runtime monitor. Uncomment to disable.
#web.submit.enable: false
#==============================================================================
# Advanced
#==============================================================================
# Override the directories for temporary files. If not specified, the
# system-specific Java temporary directory (java.io.tmpdir property) is taken.
#
# For framework setups on Yarn or Mesos, Flink will automatically pick up the
# containers' temp directories without any need for configuration.
#
# Add a delimited list for multiple directories, using the system directory
# delimiter (colon ':' on unix) or a comma, e.g.:
# /data1/tmp:/data2/tmp:/data3/tmp
#
# Note: Each directory entry is read from and written to by a different I/O
# thread. You can include the same directory multiple times in order to create
# multiple I/O threads against that directory. This is for example relevant for
# high-throughput RAIDs.
#
# io.tmp.dirs: /tmp
# The classloading resolve order. Possible values are 'child-first' (Flink's default)
# and 'parent-first' (Java's default).
#
# Child first classloading allows users to use different dependency/library
# versions in their application than those in the classpath. Switching back
# to 'parent-first' may help with debugging dependency issues.
#
# classloader.resolve-order: child-first
# The amount of memory going to the network stack. These numbers usually need
# no tuning. Adjusting them may be necessary in case of an "Insufficient number
# of network buffers" error. The default min is 64MB, the default max is 1GB.
#
# taskmanager.memory.network.fraction: 0.1
# taskmanager.memory.network.min: 64mb
# taskmanager.memory.network.max: 1gb
#==============================================================================
# Flink Cluster Security Configuration
#==============================================================================
# Kerberos authentication for various components - Hadoop, ZooKeeper, and connectors -
# may be enabled in four steps:
# 1. configure the local krb5.conf file
# 2. provide Kerberos credentials (either a keytab or a ticket cache w/ kinit)
# 3. make the credentials available to various JAAS login contexts
# 4. configure the connector to use JAAS/SASL
# The below configure how Kerberos credentials are provided. A keytab will be used instead of
# a ticket cache if the keytab path and principal are set.
# security.kerberos.login.use-ticket-cache: true
# security.kerberos.login.keytab: /path/to/kerberos/keytab
# security.kerberos.login.principal: flink-user
# The configuration below defines which JAAS login contexts
# security.kerberos.login.contexts: Client,KafkaClient
#==============================================================================
# ZK Security Configuration
#==============================================================================
# Below configurations are applicable if ZK ensemble is configured for security
# Override below configuration to provide custom ZK service name if configured
# zookeeper.sasl.service-name: zookeeper
# The configuration below must match one of the values set in "security.kerberos.login.contexts"
# zookeeper.sasl.login-context-name: Client
#==============================================================================
# HistoryServer
#==============================================================================
# The HistoryServer is started and stopped via bin/historyserver.sh (start|stop)
# Directory to upload completed jobs to. Add this directory to the list of
# monitored directories of the HistoryServer as well (see below).
#jobmanager.archive.fs.dir: hdfs:///completed-jobs/
# The address under which the web-based HistoryServer listens.
#historyserver.web.address: 0.0.0.0
# The port under which the web-based HistoryServer listens.
#historyserver.web.port: 8082
# Comma separated list of directories to monitor for completed jobs.
#historyserver.archive.fs.dir: hdfs:///completed-jobs/
# Interval in milliseconds for refreshing the monitored directories.
#historyserver.archive.fs.refresh-interval: 10000
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]During the execution of the InLong Sort-Oracle-CDC task, the Extract node generates a large number of archive logs, and the Load node fails to capture incremental change data. [inlong]
Posted by "ComplexA (via GitHub)" <gi...@apache.org>.
ComplexA commented on issue #10719:
URL: https://github.com/apache/inlong/issues/10719#issuecomment-2252352511
> @ComplexA Hello. Could you add some taskmanager log info? Maybe, you can try add table option 'debezium.log.mining.strategy' = 'online_catalog' for extract node.
Good!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]During the execution of the InLong Sort-Oracle-CDC task, the Extract node generates a large number of archive logs, and the Load node fails to capture incremental change data. [inlong]
Posted by "vernedeng (via GitHub)" <gi...@apache.org>.
vernedeng commented on issue #10719:
URL: https://github.com/apache/inlong/issues/10719#issuecomment-2249827853
@ComplexA
Would you please offer more information such as your connector options, and stream sink configuration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]During the execution of the InLong Sort-Oracle-CDC task, the Extract node generates a large number of archive logs, and the Load node fails to capture incremental change data. [inlong]
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong closed issue #10719: [Bug]During the execution of the InLong Sort-Oracle-CDC task, the Extract node generates a large number of archive logs, and the Load node fails to capture incremental change data.
URL: https://github.com/apache/inlong/issues/10719
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]During the execution of the InLong Sort-Oracle-CDC task, the Extract node generates a large number of archive logs, and the Load node fails to capture incremental change data. [inlong]
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong commented on issue #10719:
URL: https://github.com/apache/inlong/issues/10719#issuecomment-2252374921
We need execute `commit` after oracle command insert data. Otherwise, flink cdc task can't read data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]During the execution of the InLong Sort-Oracle-CDC task, the Extract node generates a large number of archive logs, and the Load node fails to capture incremental change data. [inlong]
Posted by "ComplexA (via GitHub)" <gi...@apache.org>.
ComplexA commented on issue #10719:
URL: https://github.com/apache/inlong/issues/10719#issuecomment-2250116985
> @ComplexA Maybe you haven't enable log archiving for Oracle. Refer [setup-oracle](https://inlong.apache.org/docs/data_node/extract_node/oracle-cdc#setup-oracle) to enable it
I have enable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org