You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "davidzollo (via GitHub)" <gi...@apache.org> on 2026/01/17 05:03:25 UTC
[I] [Feature][Connector] Enhance Informix Connector with CDC and Performance Optimization [seatunnel]
davidzollo opened a new issue, #10359:
URL: https://github.com/apache/seatunnel/issues/10359
## Background
IBM Informix is a robust relational database management system widely used in financial institutions, retail chains, and legacy enterprise systems, particularly in banking core systems and point-of-sale (POS) applications.
While SeaTunnel currently supports Informix through the JDBC connector with dialect support, there is a need for a **dedicated Informix connector** that provides Change Data Capture (CDC) capabilities and performance optimizations tailored to Informix-specific features.
## Motivation
- **Legacy System Integration**: Many financial institutions and retail chains still rely on Informix for mission-critical applications
- **CDC Requirements**: Real-time data replication from Informix to modern data platforms requires CDC capabilities
- **Performance Limitations**: Generic JDBC connector cannot leverage Informix-specific optimization techniques
- **Banking & Retail**: Core systems in banks and POS systems in retail require reliable, high-performance data integration
## Current Status vs. Proposed Enhancement
### Current Status (JDBC Connector)
✅ Basic read/write operations
✅ SQL query support
❌ No CDC support
❌ No Informix-specific optimizations
❌ Limited support for Informix-specific data types
❌ No smart large object (SLOB) handling
### Proposed Enhancement (Dedicated Connector)
✅ All existing JDBC capabilities
✅ **Change Data Capture (CDC)** using Informix CDC API
✅ **Performance optimizations** (fragmentation-aware queries, parallel reads)
✅ **Advanced data type support** (SERIAL8, BIGSERIAL, LVARCHAR, etc.)
✅ **Smart large object handling** (BLOB, CLOB with streaming)
✅ **Connection pooling** optimized for Informix
## Proposed Solution
Implement a dedicated Informix connector with the following capabilities:
### 1. Change Data Capture (CDC)
Use Informix's native CDC capabilities:
**Option A: CDC API (Enterprise Edition)**
- Use Informix CDC Java API
- Capture INSERT, UPDATE, DELETE operations
- Low latency change streaming
- Transaction-level consistency
**Option B: Log-Based CDC (Smart Log)**
- Read from Informix logical logs
- Parse transaction log records
- Supports Standard and Enterprise editions
**Option C: Trigger-Based CDC (Fallback)**
- Create triggers on source tables
- Write changes to shadow tables
- Compatible with all Informix versions
### 2. Performance Optimizations
- **Fragmentation-Aware Queries**: Leverage table fragmentation for parallel reads
- **Smart Buffering**: Optimize fetch size based on data types and row size
- **Connection Pooling**: Informix-specific connection pool tuning
- **Query Optimization**: Use Informix hints and optimizer directives
- **Parallel Extraction**: Multi-threaded reads aligned with fragments
### Configuration Example - CDC Mode
```hocon
source {
Informix {
# Connection
host = "informix-server.example.com"
port = 9088
database = "stores_demo"
schema = "informix"
# Authentication
username = "informix"
password = "******"
# CDC configuration
mode = "cdc" # or "snapshot", "incremental"
cdc_method = "cdc_api" # or "logical_log", "trigger_based"
# Table selection
tables = ["customer", "orders", "order_items"]
# CDC API specific (for cdc_api method)
cdc_api {
session_name = "seatunnel_cdc_session"
capture_deletes = true
capture_updates = true
capture_inserts = true
# Checkpoint configuration
checkpoint_interval_ms = 5000
start_position = "latest" # or "earliest", timestamp
}
# Performance tuning
fetch_size = 1000
max_retry_attempts = 3
connection_timeout_ms = 30000
# Data type handling
enable_slob_streaming = true
max_blob_size_mb = 10
}
}
```
### Configuration Example - Optimized Snapshot
```hocon
source {
Informix {
host = "informix-server.example.com"
port = 9088
database = "stores_demo"
schema = "informix"
username = "informix"
password = "******"
mode = "snapshot"
table = "customer"
# Performance optimization
parallel_reads = true
parallelism = 4
# Fragmentation-aware splitting
use_fragmentation_info = true
split_strategy = "by_fragment" # or "by_rowid", "by_key"
# Query optimization
use_optimizer_hints = true
optimizer_hints = "USE_NL(t1 t2)"
# Buffer tuning
fetch_size = 5000
enable_prefetch = true
socket_buffer_size_kb = 64
# Data type handling
handle_serial8_as_bigint = true
handle_lvarchar_as_string = true
}
}
```
### Configuration Example - Incremental
```hocon
source {
Informix {
host = "informix-server.example.com"
port = 9088
database = "stores_demo"
username = "informix"
password = "******"
mode = "incremental"
table = "orders"
# Incremental configuration
incremental_column = "order_date"
incremental_column_type = "datetime" # or "serial", "timestamp"
start_value = "2024-01-01 00:00:00"
# For SERIAL-based incremental
# incremental_column = "order_num"
# incremental_column_type = "serial"
# start_value = "1001"
}
}
```
### Sink Configuration Example
```hocon
sink {
Informix {
host = "informix-server.example.com"
port = 9088
database = "stores_demo"
table = "customer_copy"
username = "informix"
password = "******"
# Write mode
write_mode = "insert" # or "upsert", "update"
# For upsert mode
primary_keys = ["customer_num"]
# Batch configuration
batch_size = 1000
batch_interval_ms = 5000
# Transaction handling
enable_transaction = true
transaction_isolation = "READ_COMMITTED"
# Error handling
max_retries = 3
enable_dead_letter_queue = true
}
}
```
## Expected Benefits
1. **Real-Time Data Replication**: CDC enables near-real-time synchronization from Informix to modern data platforms
2. **Performance Improvement**: 2-5x faster than generic JDBC for large-scale data extraction
3. **Financial Compliance**: Reliable CDC for audit trails and regulatory reporting in banking systems
4. **Legacy Modernization**: Enable gradual migration from Informix to modern databases
5. **Operational Analytics**: Real-time analytics on POS and transaction data
## Technical Considerations
### Dependencies
- **Informix JDBC Driver**: IBM Informix JDBC 4.50+
- **CDC API**: Informix CDC Java API (for CDC mode)
- **Connection Pool**: HikariCP or custom pool optimized for Informix
### Informix Version Support
- **Informix 12.10+**: Full support including CDC API
- **Informix 11.70+**: Logical log-based CDC
- **Informix 11.50+**: Basic support with trigger-based CDC
### CDC Implementation
- **CDC API**: Best performance, requires Enterprise Edition
- **Logical Log Parsing**: Good performance, works with Standard Edition
- **Trigger-Based**: Fallback option, works with all versions
### Performance Considerations
- **Fragmentation**: Query fragment-specific data for parallel processing
- **Indexes**: Use index-based splits for better parallelism
- **ROWID**: Leverage ROWID for efficient data splitting
- **Smart Large Objects**: Stream BLOBs/CLOBs instead of loading into memory
### Error Handling
- **Connection Failures**: Automatic reconnection with exponential backoff
- **Transaction Rollbacks**: Retry failed transactions
- **CDC Gaps**: Detect and handle missing log records
- **Data Type Errors**: Handle Informix-specific type conversion issues
### Testing
- **Informix Developer Edition**: Free download for development and testing
- **Docker Image**: Use official IBM Informix Docker image
- **Integration Tests**: Test CDC, snapshot, and incremental modes
- **Performance Tests**: Validate fragmentation-aware parallelism
## Implementation Phases
### Phase 1: Enhanced Snapshot & Incremental (MVP)
- Fragmentation-aware parallel reads
- Optimized connection pooling
- Advanced data type support
- Performance tuning options
### Phase 2: CDC Support
- CDC API integration (Enterprise Edition)
- Logical log parsing (Standard Edition)
- Checkpoint and state management
- Exactly-once semantics
### Phase 3: Production Hardening
- Advanced error handling and retry logic
- Monitoring and metrics
- Performance profiling and optimization
- Comprehensive integration tests
### Phase 4: Advanced Features
- Multi-table CDC sessions
- Schema evolution handling
- Compression and encryption support
- Integration with Informix High Availability (HDR)
## References
- [Informix Change Data Capture](https://www.ibm.com/docs/en/informix-servers/14.10?topic=capture-change-data)
- [Informix JDBC Driver Documentation](https://www.ibm.com/docs/en/informix-servers/14.10?topic=drivers-jdbc-driver)
- [Informix Data Types](https://www.ibm.com/docs/en/informix-servers/14.10?topic=types-built-in-data)
- [Informix Performance Guide](https://www.ibm.com/docs/en/informix-servers/14.10?topic=guide-performance)
## Community Impact
This connector will:
- Enable financial institutions to modernize their data infrastructure while maintaining Informix systems
- Provide real-time analytics capabilities for retail POS systems
- Support regulatory compliance and audit requirements in banking
- Position SeaTunnel as a viable solution for legacy system integration
---
**Priority**: Medium
**Estimated Effort**: Medium
**Target Release**: 2.3.15 or 3.0.0
**Note**: This builds upon existing JDBC dialect support
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Connector] Add Informix CDC Connector [seatunnel]
Posted by "DanielCarter-stack (via GitHub)" <gi...@apache.org>.
DanielCarter-stack commented on issue #10359:
URL: https://github.com/apache/seatunnel/issues/10359#issuecomment-3764667896
<!-- code-pr-reviewer -->
Thanks for the proposalfeature request. Before we can implement an Informix CDC connector, we need to clarify a few points:
1. **Debezium support**: Does Debezium have an official Informix connector? Based on our review, Debezium's supported connectors currently include MySQL, PostgreSQL, Oracle, SQL Server, and others, but Informix is not listed. Could you confirm if there's a Debezium Informix connector available, or are you proposing a custom CDC implementation?
2. **Reference information**: Could you provide:
- Informix version(s) you intend to support
- Any existing CDC solutions for Informix (e.g., built-in CDC logs, Change Data Capture features)
- Documentation links for Informix CDC mechanisms
3. **Architecture preference**: Are you expecting this to follow the same pattern as existing SeaTunnel CDC connectors (e.g., `seatunnel-connectors-v2/connector-cdc-mysql`), which rely on Debezium?
Without a clear CDC mechanism from Informix or Debezium support, we cannot proceed with the implementation. Please share the above details so we can evaluate feasibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Connector] Add Informix CDC Connector [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #10359:
URL: https://github.com/apache/seatunnel/issues/10359#issuecomment-4203452851
I re-checked the repository state for this proposal. At the moment, SeaTunnel does recognize Informix on the JDBC dialect side, but I do not see a standalone Informix-CDC connector/module in the current repo. So the gap described here is real: JDBC support should not be interpreted as CDC support.
I also do not see a dev-branch PR linked to this issue yet. My suggestion would be to keep this issue as the feature entry point, but narrow the first implementation slice very explicitly: source first, snapshot + CDC semantics first, and a clear decision on which Informix CDC mechanism is actually feasible for an MVP.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Connector] Add Informix CDC Connector [seatunnel]
Posted by "DanielCarter-stack (via GitHub)" <gi...@apache.org>.
DanielCarter-stack commented on issue #10359:
URL: https://github.com/apache/seatunnel/issues/10359#issuecomment-3763720670
<!-- code-pr-reviewer -->
Thanks for the proposal. Could you clarify a few things to help move this forward?
1. Which Informix CDC source should the connector consume? (e.g., built-in CDC logs, a specific change data capture method, or a third-party tool like Debezium?)
2. What ingestion format do you expect? (Change logs per row, or per-table/micro-batch?)
3. Any constraints or preferences for the client library/protocol? (e.g., JDBC-only, native wire protocol, or an external CDC agent?)
For reference, existing CDC connectors (e.g., `seatunnel-connectors-v2/connector-cdc`) follow patterns in `connector-cdc-base` and often delegate to external CDC pipelines rather than polling database logs directly. Providing these details will help evaluate feasibility and identify the right integration approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Connector] Add Informix CDC Connector [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on issue #10359:
URL: https://github.com/apache/seatunnel/issues/10359#issuecomment-5446898531
@davidzollo @DanielLeens I check the latest dev and was checking the feasibility now.
The feasibility has changed since the earlier discussion. Debezium now has an incubating Informix connector built on the IBM Change Streams API, so we would not need to implement a separate log parser. It supports initial snapshot and incremental streaming and looks like the most maintainable base for this connector.
The blocker is version compatibility. SeaTunnel CDC currently uses Debezium 1.9.8 and targets Java 8. Informix support was introduced in Debezium 2.5, which requires Java 11. Adding only the Informix artifact would mix incompatible Debezium APIs and Java bytecode levels.
I do not think backporting the connector or maintaining another Informix log reader would be a good long-term solution. Before starting implementation, can we agree on one of these directions:
1. Informix CDC waits for the SeaTunnel Debezium and Java baseline upgrade
2. Informix runs through an isolated Java 11+ connector runtime
3. the community prefers a custom IBM Change Streams adapter
My preference is the first option, or the second if SeaTunnel wants an isolated runtime boundary. Once the direction is confirmed I can prepare a compatibility spike and define the source-only snapshot plus CDC MVP. Let me know. thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Connector] Add Informix CDC Connector [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on issue #10359:
URL: https://github.com/apache/seatunnel/issues/10359#issuecomment-3765289029
Please leave a message if you're willing to implement this feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Connector] Add Informix CDC Connector [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #10359:
URL: https://github.com/apache/seatunnel/issues/10359#issuecomment-5453928407
Thanks for rechecking the landscape.
I rechecked current `dev` as of commit `1ff72ceb3d2a823a0fca64ec009984d75e846c28`. I still do not see an Informix CDC connector/module in-tree, and I also do not see an open PR that already covers this feature.
Given that, your compatibility summary looks directionally right to me: the main blocker is not log parsing anymore, it is the SeaTunnel baseline mismatch with the Debezium / Java level needed by the newer Informix connector. From a maintainer perspective, option 1 is the cleanest default path, option 2 is a much larger runtime-boundary discussion, and option 3 should be the fallback only if the first two are rejected.
So I would treat this issue as blocked on baseline direction rather than ready for implementation immediately. If you want to keep moving, a short compatibility spike would be useful: exact Debezium 2.5+ touchpoints, Java 11+ impact, and the smallest source-only snapshot + CDC MVP we could review once the baseline decision is clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org