You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "davidzollo (via GitHub)" <gi...@apache.org> on 2026/01/17 05:03:25 UTC

[I] [Feature][Connector] Enhance Informix Connector with CDC and Performance Optimization [seatunnel]

davidzollo opened a new issue, #10359:
URL: https://github.com/apache/seatunnel/issues/10359

   ## Background
   
   IBM Informix is a robust relational database management system widely used in financial institutions, retail chains, and legacy enterprise systems, particularly in banking core systems and point-of-sale (POS) applications.
   
   While SeaTunnel currently supports Informix through the JDBC connector with dialect support, there is a need for a **dedicated Informix connector** that provides Change Data Capture (CDC) capabilities and performance optimizations tailored to Informix-specific features.
   
   ## Motivation
   
   - **Legacy System Integration**: Many financial institutions and retail chains still rely on Informix for mission-critical applications
   - **CDC Requirements**: Real-time data replication from Informix to modern data platforms requires CDC capabilities
   - **Performance Limitations**: Generic JDBC connector cannot leverage Informix-specific optimization techniques
   - **Banking & Retail**: Core systems in banks and POS systems in retail require reliable, high-performance data integration
   
   ## Current Status vs. Proposed Enhancement
   
   ### Current Status (JDBC Connector)
   ✅ Basic read/write operations  
   ✅ SQL query support  
   ❌ No CDC support  
   ❌ No Informix-specific optimizations  
   ❌ Limited support for Informix-specific data types  
   ❌ No smart large object (SLOB) handling  
   
   ### Proposed Enhancement (Dedicated Connector)
   ✅ All existing JDBC capabilities  
   ✅ **Change Data Capture (CDC)** using Informix CDC API  
   ✅ **Performance optimizations** (fragmentation-aware queries, parallel reads)  
   ✅ **Advanced data type support** (SERIAL8, BIGSERIAL, LVARCHAR, etc.)  
   ✅ **Smart large object handling** (BLOB, CLOB with streaming)  
   ✅ **Connection pooling** optimized for Informix  
   
   ## Proposed Solution
   
   Implement a dedicated Informix connector with the following capabilities:
   
   ### 1. Change Data Capture (CDC)
   
   Use Informix's native CDC capabilities:
   
   **Option A: CDC API (Enterprise Edition)**
   - Use Informix CDC Java API
   - Capture INSERT, UPDATE, DELETE operations
   - Low latency change streaming
   - Transaction-level consistency
   
   **Option B: Log-Based CDC (Smart Log)**
   - Read from Informix logical logs
   - Parse transaction log records
   - Supports Standard and Enterprise editions
   
   **Option C: Trigger-Based CDC (Fallback)**
   - Create triggers on source tables
   - Write changes to shadow tables
   - Compatible with all Informix versions
   
   ### 2. Performance Optimizations
   
   - **Fragmentation-Aware Queries**: Leverage table fragmentation for parallel reads
   - **Smart Buffering**: Optimize fetch size based on data types and row size
   - **Connection Pooling**: Informix-specific connection pool tuning
   - **Query Optimization**: Use Informix hints and optimizer directives
   - **Parallel Extraction**: Multi-threaded reads aligned with fragments
   
   ### Configuration Example - CDC Mode
   
   ```hocon
   source {
     Informix {
       # Connection
       host = "informix-server.example.com"
       port = 9088
       database = "stores_demo"
       schema = "informix"
       
       # Authentication
       username = "informix"
       password = "******"
       
       # CDC configuration
       mode = "cdc" # or "snapshot", "incremental"
       cdc_method = "cdc_api" # or "logical_log", "trigger_based"
       
       # Table selection
       tables = ["customer", "orders", "order_items"]
       
       # CDC API specific (for cdc_api method)
       cdc_api {
         session_name = "seatunnel_cdc_session"
         capture_deletes = true
         capture_updates = true
         capture_inserts = true
         
         # Checkpoint configuration
         checkpoint_interval_ms = 5000
         start_position = "latest" # or "earliest", timestamp
       }
       
       # Performance tuning
       fetch_size = 1000
       max_retry_attempts = 3
       connection_timeout_ms = 30000
       
       # Data type handling
       enable_slob_streaming = true
       max_blob_size_mb = 10
     }
   }
   ```
   
   ### Configuration Example - Optimized Snapshot
   
   ```hocon
   source {
     Informix {
       host = "informix-server.example.com"
       port = 9088
       database = "stores_demo"
       schema = "informix"
       username = "informix"
       password = "******"
       
       mode = "snapshot"
       table = "customer"
       
       # Performance optimization
       parallel_reads = true
       parallelism = 4
       
       # Fragmentation-aware splitting
       use_fragmentation_info = true
       split_strategy = "by_fragment" # or "by_rowid", "by_key"
       
       # Query optimization
       use_optimizer_hints = true
       optimizer_hints = "USE_NL(t1 t2)"
       
       # Buffer tuning
       fetch_size = 5000
       enable_prefetch = true
       socket_buffer_size_kb = 64
       
       # Data type handling
       handle_serial8_as_bigint = true
       handle_lvarchar_as_string = true
     }
   }
   ```
   
   ### Configuration Example - Incremental
   
   ```hocon
   source {
     Informix {
       host = "informix-server.example.com"
       port = 9088
       database = "stores_demo"
       username = "informix"
       password = "******"
       
       mode = "incremental"
       table = "orders"
       
       # Incremental configuration
       incremental_column = "order_date"
       incremental_column_type = "datetime" # or "serial", "timestamp"
       start_value = "2024-01-01 00:00:00"
       
       # For SERIAL-based incremental
       # incremental_column = "order_num"
       # incremental_column_type = "serial"
       # start_value = "1001"
     }
   }
   ```
   
   ### Sink Configuration Example
   
   ```hocon
   sink {
     Informix {
       host = "informix-server.example.com"
       port = 9088
       database = "stores_demo"
       table = "customer_copy"
       username = "informix"
       password = "******"
       
       # Write mode
       write_mode = "insert" # or "upsert", "update"
       
       # For upsert mode
       primary_keys = ["customer_num"]
       
       # Batch configuration
       batch_size = 1000
       batch_interval_ms = 5000
       
       # Transaction handling
       enable_transaction = true
       transaction_isolation = "READ_COMMITTED"
       
       # Error handling
       max_retries = 3
       enable_dead_letter_queue = true
     }
   }
   ```
   
   ## Expected Benefits
   
   1. **Real-Time Data Replication**: CDC enables near-real-time synchronization from Informix to modern data platforms
   2. **Performance Improvement**: 2-5x faster than generic JDBC for large-scale data extraction
   3. **Financial Compliance**: Reliable CDC for audit trails and regulatory reporting in banking systems
   4. **Legacy Modernization**: Enable gradual migration from Informix to modern databases
   5. **Operational Analytics**: Real-time analytics on POS and transaction data
   
   ## Technical Considerations
   
   ### Dependencies
   - **Informix JDBC Driver**: IBM Informix JDBC 4.50+
   - **CDC API**: Informix CDC Java API (for CDC mode)
   - **Connection Pool**: HikariCP or custom pool optimized for Informix
   
   ### Informix Version Support
   - **Informix 12.10+**: Full support including CDC API
   - **Informix 11.70+**: Logical log-based CDC
   - **Informix 11.50+**: Basic support with trigger-based CDC
   
   ### CDC Implementation
   - **CDC API**: Best performance, requires Enterprise Edition
   - **Logical Log Parsing**: Good performance, works with Standard Edition
   - **Trigger-Based**: Fallback option, works with all versions
   
   ### Performance Considerations
   - **Fragmentation**: Query fragment-specific data for parallel processing
   - **Indexes**: Use index-based splits for better parallelism
   - **ROWID**: Leverage ROWID for efficient data splitting
   - **Smart Large Objects**: Stream BLOBs/CLOBs instead of loading into memory
   
   ### Error Handling
   - **Connection Failures**: Automatic reconnection with exponential backoff
   - **Transaction Rollbacks**: Retry failed transactions
   - **CDC Gaps**: Detect and handle missing log records
   - **Data Type Errors**: Handle Informix-specific type conversion issues
   
   ### Testing
   - **Informix Developer Edition**: Free download for development and testing
   - **Docker Image**: Use official IBM Informix Docker image
   - **Integration Tests**: Test CDC, snapshot, and incremental modes
   - **Performance Tests**: Validate fragmentation-aware parallelism
   
   ## Implementation Phases
   
   ### Phase 1: Enhanced Snapshot & Incremental (MVP)
   - Fragmentation-aware parallel reads
   - Optimized connection pooling
   - Advanced data type support
   - Performance tuning options
   
   ### Phase 2: CDC Support
   - CDC API integration (Enterprise Edition)
   - Logical log parsing (Standard Edition)
   - Checkpoint and state management
   - Exactly-once semantics
   
   ### Phase 3: Production Hardening
   - Advanced error handling and retry logic
   - Monitoring and metrics
   - Performance profiling and optimization
   - Comprehensive integration tests
   
   ### Phase 4: Advanced Features
   - Multi-table CDC sessions
   - Schema evolution handling
   - Compression and encryption support
   - Integration with Informix High Availability (HDR)
   
   ## References
   
   - [Informix Change Data Capture](https://www.ibm.com/docs/en/informix-servers/14.10?topic=capture-change-data)
   - [Informix JDBC Driver Documentation](https://www.ibm.com/docs/en/informix-servers/14.10?topic=drivers-jdbc-driver)
   - [Informix Data Types](https://www.ibm.com/docs/en/informix-servers/14.10?topic=types-built-in-data)
   - [Informix Performance Guide](https://www.ibm.com/docs/en/informix-servers/14.10?topic=guide-performance)
   
   ## Community Impact
   
   This connector will:
   - Enable financial institutions to modernize their data infrastructure while maintaining Informix systems
   - Provide real-time analytics capabilities for retail POS systems
   - Support regulatory compliance and audit requirements in banking
   - Position SeaTunnel as a viable solution for legacy system integration
   
   ---
   
   **Priority**: Medium  
   **Estimated Effort**: Medium  
   **Target Release**: 2.3.15 or 3.0.0  
   **Note**: This builds upon existing JDBC dialect support


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector] Add Informix CDC Connector [seatunnel]

Posted by "DanielCarter-stack (via GitHub)" <gi...@apache.org>.
DanielCarter-stack commented on issue #10359:
URL: https://github.com/apache/seatunnel/issues/10359#issuecomment-3764667896

   <!-- code-pr-reviewer -->
   Thanks for the proposalfeature request. Before we can implement an Informix CDC connector, we need to clarify a few points:
   
   1. **Debezium support**: Does Debezium have an official Informix connector? Based on our review, Debezium's supported connectors currently include MySQL, PostgreSQL, Oracle, SQL Server, and others, but Informix is not listed. Could you confirm if there's a Debezium Informix connector available, or are you proposing a custom CDC implementation?
   
   2. **Reference information**: Could you provide:
      - Informix version(s) you intend to support
      - Any existing CDC solutions for Informix (e.g., built-in CDC logs, Change Data Capture features)
      - Documentation links for Informix CDC mechanisms
   
   3. **Architecture preference**: Are you expecting this to follow the same pattern as existing SeaTunnel CDC connectors (e.g., `seatunnel-connectors-v2/connector-cdc-mysql`), which rely on Debezium?
   
   Without a clear CDC mechanism from Informix or Debezium support, we cannot proceed with the implementation. Please share the above details so we can evaluate feasibility.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector] Add Informix CDC Connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #10359:
URL: https://github.com/apache/seatunnel/issues/10359#issuecomment-4203452851

   I re-checked the repository state for this proposal. At the moment, SeaTunnel does recognize Informix on the JDBC dialect side, but I do not see a standalone Informix-CDC connector/module in the current repo. So the gap described here is real: JDBC support should not be interpreted as CDC support.
   
   I also do not see a dev-branch PR linked to this issue yet. My suggestion would be to keep this issue as the feature entry point, but narrow the first implementation slice very explicitly: source first, snapshot + CDC semantics first, and a clear decision on which Informix CDC mechanism is actually feasible for an MVP.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector] Add Informix CDC Connector [seatunnel]

Posted by "DanielCarter-stack (via GitHub)" <gi...@apache.org>.
DanielCarter-stack commented on issue #10359:
URL: https://github.com/apache/seatunnel/issues/10359#issuecomment-3763720670

   <!-- code-pr-reviewer -->
   Thanks for the proposal. Could you clarify a few things to help move this forward?
   
   1. Which Informix CDC source should the connector consume? (e.g., built-in CDC logs, a specific change data capture method, or a third-party tool like Debezium?)
   2. What ingestion format do you expect? (Change logs per row, or per-table/micro-batch?)
   3. Any constraints or preferences for the client library/protocol? (e.g., JDBC-only, native wire protocol, or an external CDC agent?)
   
   For reference, existing CDC connectors (e.g., `seatunnel-connectors-v2/connector-cdc`) follow patterns in `connector-cdc-base` and often delegate to external CDC pipelines rather than polling database logs directly. Providing these details will help evaluate feasibility and identify the right integration approach.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector] Add Informix CDC Connector [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on issue #10359:
URL: https://github.com/apache/seatunnel/issues/10359#issuecomment-5446898531

   @davidzollo @DanielLeens I check the latest dev and was checking the feasibility now. 
   
   The feasibility has changed since the earlier discussion. Debezium now has an incubating Informix connector built on the IBM Change Streams API, so we would not need to implement a separate log parser. It supports initial snapshot and incremental streaming and looks like the most maintainable base for this connector.
   
   The blocker is version compatibility. SeaTunnel CDC currently uses Debezium 1.9.8 and targets Java 8. Informix support was introduced in Debezium 2.5, which requires Java 11. Adding only the Informix artifact would mix incompatible Debezium APIs and Java bytecode levels.
   
   I do not think backporting the connector or maintaining another Informix log reader would be a good long-term solution. Before starting implementation, can we agree on one of these directions:
   
   1. Informix CDC waits for the SeaTunnel Debezium and Java baseline upgrade
   2. Informix runs through an isolated Java 11+ connector runtime
   3. the community prefers a custom IBM Change Streams adapter
   
   My preference is the first option, or the second if SeaTunnel wants an isolated runtime boundary. Once the direction is confirmed I can prepare a compatibility spike and define the source-only snapshot plus CDC MVP. Let me know. thanks! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector] Add Informix CDC Connector [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on issue #10359:
URL: https://github.com/apache/seatunnel/issues/10359#issuecomment-3765289029

   Please leave a message if you're willing to implement this feature


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector] Add Informix CDC Connector [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #10359:
URL: https://github.com/apache/seatunnel/issues/10359#issuecomment-5453928407

   Thanks for rechecking the landscape.
   
   I rechecked current `dev` as of commit `1ff72ceb3d2a823a0fca64ec009984d75e846c28`. I still do not see an Informix CDC connector/module in-tree, and I also do not see an open PR that already covers this feature.
   
   Given that, your compatibility summary looks directionally right to me: the main blocker is not log parsing anymore, it is the SeaTunnel baseline mismatch with the Debezium / Java level needed by the newer Informix connector. From a maintainer perspective, option 1 is the cleanest default path, option 2 is a much larger runtime-boundary discussion, and option 3 should be the fallback only if the first two are rejected.
   
   So I would treat this issue as blocked on baseline direction rather than ready for implementation immediately. If you want to keep moving, a short compatibility spike would be useful: exact Debezium 2.5+ touchpoints, Java 11+ impact, and the smallest source-only snapshot + CDC MVP we could review once the baseline decision is clear.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org