You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "tusharagrahari (via GitHub)" <gi...@apache.org> on 2026/06/25 20:56:26 UTC
[PR] feat(connectors): add MySQL source polling connector [iggy]
tusharagrahari opened a new pull request, #3568:
URL: https://github.com/apache/iggy/pull/3568
Which issue does this PR address?
Relates to #3445
Rationale
MySQL is a widely used relational database with no existing Iggy source connector. This adds incremental table polling so users can stream MySQL rows into Iggy topics without CDC infrastructure.
What changed?
Before this PR, there was no way to source data from MySQL into Iggy.
This adds a mysql_source connector plugin supporting incremental table polling via a configurable tracking column. Rows are streamed as JSON, raw bytes, or text depending on the configured payload format. Post-processing options (delete after read, mark as processed) and custom SQL queries with parameter substitution are also supported. CDC (binlog-based) is out of scope and planned as a follow-up.
Local Execution
- Passed
- Pre-commit hooks ran
AI Usage
1. Tools: Claude (Claude Code)
2. Scope: Writing unit and integration tests, thinking through edge cases, and pair-programming during development.
3. Verification: Ran all tests locally against a real MySQL container, validated end-to-end polling flow, and ran the full pre-commit hook suite.
4. Yes, I can explain every line of code if asked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3603840349
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1563 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ // Skip the pacing delay while draining a backlog
+ if !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ // Idle cycles produce no messages and leave offsets/processed_rows
+ // untouched, so return None to let the runtime skip the state fsync.
+ let persisted_state = if messages.is_empty() {
+ None
+ } else {
+ self.serialize_state(&state)
+ };
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+ let offset = format_offset_value(&offset_value);
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now_unix", &now.timestamp().to_string())
+ .replace("$now", &now.to_rfc3339())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
Review Comment:
BINARY PK base64 in JSON path; IN clause uses quoted string literal ⇒ mark/delete miss. **Fix**: hex/binary bind or typed PK encoder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3509626715
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
Review Comment:
delete_after_read / processed_column commit their DELETE / UPDATE to mysql during poll, before the runtime has sent the batch to iggy (the runtime persists connector state only after a successful send, and the `Source` trait has no ack path). a send failure or crash in that window means rows destroyed or marked but never delivered - these modes are effectively at-most-once.
this can't be fixed connector-side without a runtime send-ack, so at minimum the readme should document both modes as at-most-once. the sink side of this gap is already tracked (#2927, #2928) and #2940 discusses the contract - the source side needs the sibling issue (postgres_source has the same behavior).
##########
core/connectors/sources/mysql_source/README.md:
##########
@@ -0,0 +1,382 @@
+# MySQL Source Connector
+
+The MySQL source connector fetches data from MySQL databases and streams it to Iggy topics. It supports incremental table polling with flexible payload extraction.
+
+> **Note:** Only polling mode is available. Binlog CDC support is planned for a future release.
+
+## Features
+
+- **Table Polling**: Incrementally fetch data from MySQL tables using a tracking column
+- **Flexible Payload Extraction**: Extract BLOB, TEXT, or JSON columns directly as payload
+- **Custom Queries**: Use custom SQL queries with parameter substitution
+- **Delete After Read**: Automatically delete rows after processing
+- **Mark as Processed**: Mark rows as processed using a boolean column
+- **Multiple Tables**: Monitor multiple tables simultaneously
+- **Batch Processing**: Fetch data in configurable batch sizes
+- **Offset Tracking**: Keep track of processed records to avoid duplicates
+
+## Configuration
+
+```toml
+[plugin_config]
+connection_string = "mysql://user:pass@localhost:3306/database"
+tables = ["users", "orders"]
+poll_interval = "1s"
+batch_size = 1000
+tracking_column = "id"
+initial_offset = "0"
+max_connections = 10
+snake_case_columns = false
+include_metadata = true
+
+# Payload extraction (optional)
+payload_column = "payload"
+payload_format = "bytea"
+
+# Delete/mark processed (optional)
+delete_after_read = false
+processed_column = "is_processed"
+primary_key_column = "id"
+
+# Custom query (optional)
+custom_query = "SELECT * FROM $table WHERE id > $offset ORDER BY id LIMIT $limit"
+```
+
+## Configuration Options
+
+| Option | Type | Default | Description |
+| ------ | ---- | ------- | ----------- |
+| `connection_string` | string | required | MySQL connection string (`mysql://user:pass@host:3306/db`) |
+| `tables` | array | required | List of tables to monitor |
+| `poll_interval` | string | `10s` | How often to poll (e.g., `1s`, `5m`) |
+| `batch_size` | u32 | `1000` | Max rows per poll |
+| `tracking_column` | string | `id` | Column for incremental updates |
+| `initial_offset` | string | none | Starting value for tracking column |
+| `max_connections` | u32 | `10` | Max database connections |
+| `snake_case_columns` | bool | `false` | Convert column names to snake_case |
+| `include_metadata` | bool | `true` | Wrap results with metadata envelope |
+| `payload_column` | string | none | Column to extract directly as payload |
+| `payload_format` | string | `bytea` | Format of payload_column: `bytea`, `text`, or `json_direct` |
Review Comment:
default is wrong: with `payload_column` set and `payload_format` unset, the format resolves to json and `open()` rejects that combination with an init error, so there is no `bytea` default - the field is effectively required. config.toml gets this right already ("required when payload_column is set").
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
Review Comment:
`$offset` is substituted raw here while the built query path escapes it via `format_offset_value`. the offset value is the previous batch's tracking-column content, i.e. database row data - with a string tracking column, hostile row content becomes SQL on the next poll (second-order injection). escape it the same way as the built path, and scope the readme's "SQL injection protection" section to the built query path.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
Review Comment:
strict `>` plus LIMIT loses rows when more than `batch_size` rows share one tracking value (routine with a second-resolution timestamp tracking column): the first poll returns `batch_size` of them and sets the offset to that value, the next poll's `>` excludes the rest forever. unique auto-increment ids are safe; the documented timestamp-tracking setup is not. options: `>=` with a pk tiebreak, or document that the tracking column must be unique.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).execute(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. A table's messages are emitted
+ // only after its rows are marked or deleted, so a failure here can never
+ // leave a table marked-but-unpublished. `mark_or_delete_processed_rows`
+ // retries transient errors; a permanent failure isolates that one table
+ // (its rows stay in MySQL and are retried next cycle) while the others
+ // still flow.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if !batch.processed_ids.is_empty()
+ && let Err(error) = self
+ .mark_or_delete_processed_rows(
+ pool,
+ &batch.table,
+ pk_column,
+ &batch.processed_ids,
+ )
+ .await
+ {
+ error!(
+ "Failed to mark or delete processed rows for table '{}', skipping this cycle: {error}",
+ batch.table
+ );
+ continue;
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ state_updates.push((batch.table, offset));
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
+ }
+ state.last_poll_time = Utc::now();
+ }
+
+ Ok(messages)
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, Error> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.validate_custom_query(custom_query)?;
+ self.substitute_query_params(custom_query, table, last_offset, batch_size)
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ for row in rows {
+ let processed = self.process_row(&row, &table_config)?;
Review Comment:
a deterministically failing row (non-utf8 in `text` format, non-json in `json_direct`, type mismatch) aborts the whole table batch via `?`, the offset never advances, and the same row re-fails every cycle - the table stalls forever with only a per-cycle error log. if that's the intended operator contract, document it in the readme; a per-row skip/dead-letter would be friendlier.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
Review Comment:
this runs per table per poll (with a full `to_uppercase` copy of the query) for config that never changes - validate once in `open()`. that also turns the `$table`-with-no-tables case into a startup error instead of a silently skipped table, and gives a place to catch the empty-`$offset` case.
##########
core/connectors/sources/mysql_source/README.md:
##########
@@ -0,0 +1,382 @@
+# MySQL Source Connector
+
+The MySQL source connector fetches data from MySQL databases and streams it to Iggy topics. It supports incremental table polling with flexible payload extraction.
+
+> **Note:** Only polling mode is available. Binlog CDC support is planned for a future release.
+
+## Features
+
+- **Table Polling**: Incrementally fetch data from MySQL tables using a tracking column
+- **Flexible Payload Extraction**: Extract BLOB, TEXT, or JSON columns directly as payload
+- **Custom Queries**: Use custom SQL queries with parameter substitution
+- **Delete After Read**: Automatically delete rows after processing
+- **Mark as Processed**: Mark rows as processed using a boolean column
+- **Multiple Tables**: Monitor multiple tables simultaneously
+- **Batch Processing**: Fetch data in configurable batch sizes
+- **Offset Tracking**: Keep track of processed records to avoid duplicates
Review Comment:
"avoid duplicates" oversells - the delete/mark modes are at-most-once (rows mutated before the iggy send is confirmed), and offset mode can either lose a batch on a live send failure or replay after a crash. a short delivery-semantics section would set expectations honestly.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
Review Comment:
`jsonb`/`jsonb_direct` are postgres-isms - mysql has no jsonb. harmless, but accepting them (and the unit test locking them in) just confuses; suggest dropping the aliases.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).execute(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. A table's messages are emitted
+ // only after its rows are marked or deleted, so a failure here can never
+ // leave a table marked-but-unpublished. `mark_or_delete_processed_rows`
Review Comment:
this guarantee only holds within a single poll - a crash or send failure after the mark/delete commits leaves exactly a marked-but-unpublished table (the messages are published by the runtime after poll returns). reword to scope it to the mark-call failure case, otherwise it reads as a stronger guarantee than the code provides.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
Review Comment:
the sleep is unconditional, so a backlogged table drains at `batch_size / poll_interval` max (defaults: 1000 rows / 10s = 100 rows/s) - initial backfills crawl. common pattern is to skip the sleep when the previous batch came back full.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).execute(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. A table's messages are emitted
+ // only after its rows are marked or deleted, so a failure here can never
+ // leave a table marked-but-unpublished. `mark_or_delete_processed_rows`
+ // retries transient errors; a permanent failure isolates that one table
+ // (its rows stay in MySQL and are retried next cycle) while the others
+ // still flow.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if !batch.processed_ids.is_empty()
+ && let Err(error) = self
+ .mark_or_delete_processed_rows(
+ pool,
+ &batch.table,
+ pk_column,
+ &batch.processed_ids,
+ )
+ .await
+ {
+ error!(
+ "Failed to mark or delete processed rows for table '{}', skipping this cycle: {error}",
+ batch.table
+ );
+ continue;
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ state_updates.push((batch.table, offset));
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
+ }
+ state.last_poll_time = Utc::now();
+ }
+
+ Ok(messages)
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, Error> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.validate_custom_query(custom_query)?;
+ self.substitute_query_params(custom_query, table, last_offset, batch_size)
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ for row in rows {
+ let processed = self.process_row(&row, &table_config)?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+ if let Some(offset) = processed.max_offset {
+ batch.max_offset = Some(offset);
+ }
+
+ batch.messages.push(processed.message);
+ }
+
+ Ok(batch)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<String> = None;
+ let mut max_offset: Option<String> = None;
+ let mut extracted_payload: Option<Vec<u8>> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ if name == config.payload_col {
+ extracted_payload =
+ Some(self.extract_payload_column(row, i, config.payload_format)?);
+ }
+ if name == config.tracking_column {
+ max_offset = value_as_string(&extract_column_value(row, i)?);
+ }
+ if name == config.pk_column {
+ row_pk = value_as_string(&extract_column_value(row, i)?);
+ }
+ }
+ }
+
+ if extracted_payload.is_none() {
+ let mut data = serde_json::Map::new();
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ let column_name = if config.snake_case_columns {
+ to_snake_case(name)
+ } else {
+ name.to_string()
+ };
+ let value = extract_column_value(row, i)?;
+ if name == config.tracking_column {
+ max_offset = value_as_string(&value);
+ }
+ if name == config.pk_column {
+ row_pk = value_as_string(&value);
+ }
+ data.insert(column_name, value);
+ }
+
+ extracted_payload = Some(if config.include_metadata {
+ let record = DatabaseRecord {
+ table_name: config.table.to_string(),
+ operation_type: "SELECT".to_string(),
+ timestamp: Utc::now(),
+ data: serde_json::Value::Object(data),
+ old_data: None,
+ };
+ simd_json::to_vec(&record).map_err(|_| Error::InvalidRecord)?
+ } else {
+ simd_json::to_vec(&data).map_err(|_| Error::InvalidRecord)?
+ });
+ }
+
+ // Both paths above always assign extracted_payload before reaching here.
+ Ok(build_processed_row(
+ extracted_payload.unwrap(),
+ max_offset,
+ row_pk,
+ ))
+ }
+}
+
+fn extract_column_value(row: &MySqlRow, column_index: usize) -> Result<serde_json::Value, Error> {
+ let column = &row.columns()[column_index];
+ let type_name = column.type_info().name();
+
+ match type_name {
+ "BOOLEAN" => {
+ let value: Option<bool> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::Bool)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT" => {
+ let value: Option<i8> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT" => {
+ let value: Option<i16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT" | "INT" => {
+ let value: Option<i32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT" => {
+ let value: Option<i64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT UNSIGNED" => {
+ let value: Option<u8> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT UNSIGNED" => {
+ let value: Option<u16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT UNSIGNED" | "INT UNSIGNED" => {
+ let value: Option<u32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT UNSIGNED" => {
+ let value: Option<u64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "FLOAT" => {
+ let value: Option<f32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as f64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DOUBLE" => {
+ let value: Option<f64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DECIMAL" => {
Review Comment:
`DECIMAL` columns can never be polled. sqlx checks `T::compatible(type_info)` before decoding, and `String` is not compatible with decimal types (sqlx-mysql accepts only the varchar/blob/enum family for strings), so `try_get::<Option<String>>` fails on every row. the workspace sqlx dep has neither `bigdecimal` nor `rust_decimal` features enabled, so there is currently no compatible type to decode into at all. since a single row error aborts the whole table batch (line 621) and the offset never advances, any table with a DECIMAL column stalls forever - and the readme documents DECIMAL as supported.
fix: enable the `rust_decimal` (or `bigdecimal`) sqlx feature and decode via that + `to_string()`, or use `try_get_unchecked` (mysql sends DECIMAL as ascii text on the wire). none of the test tables has a DECIMAL column, which is why CI is green - worth adding one.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
Review Comment:
custom query with `$offset` but no stored offset and no `initial_offset` substitutes an empty string, so `WHERE id > $offset` becomes `WHERE id > ` - invalid SQL that fails every poll, and since no rows are ever fetched no offset ever gets stored, so it never recovers. the custom-query example in the readme is exactly this shape. validate in `open()`: require `initial_offset` when the query contains a bare `$offset`.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
Review Comment:
`.replace("$now", ...)` runs before `.replace("$now_unix", ...)` and matches the `$now` prefix inside `$now_unix`, turning it into `<rfc3339>_unix` - the documented `$now_unix` placeholder can never work and produces invalid SQL. swap the two replaces (longest token first).
the unit test at 1258-1268 stays green because it only asserts the tokens are absent, which the mangled output also satisfies - make it assert the substituted values are actually present.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).execute(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. A table's messages are emitted
+ // only after its rows are marked or deleted, so a failure here can never
+ // leave a table marked-but-unpublished. `mark_or_delete_processed_rows`
+ // retries transient errors; a permanent failure isolates that one table
+ // (its rows stay in MySQL and are retried next cycle) while the others
+ // still flow.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if !batch.processed_ids.is_empty()
+ && let Err(error) = self
+ .mark_or_delete_processed_rows(
+ pool,
+ &batch.table,
+ pk_column,
+ &batch.processed_ids,
+ )
+ .await
+ {
+ error!(
+ "Failed to mark or delete processed rows for table '{}', skipping this cycle: {error}",
+ batch.table
+ );
+ continue;
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ state_updates.push((batch.table, offset));
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
Review Comment:
tracking offsets advance in memory at the end of every poll regardless of whether the runtime managed to send the batch. the connector never re-reads persisted state while running, so a transient send failure without a crash means the next poll queries past the failed batch and those rows are never read again. crash+restart recovers (disk state is older) - only the live send-fail path loses data. same runtime-ack limitation as the delete_after_read comment - source-side sibling of #2927/#2928, same follow-up issue.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).execute(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. A table's messages are emitted
+ // only after its rows are marked or deleted, so a failure here can never
+ // leave a table marked-but-unpublished. `mark_or_delete_processed_rows`
+ // retries transient errors; a permanent failure isolates that one table
+ // (its rows stay in MySQL and are retried next cycle) while the others
+ // still flow.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if !batch.processed_ids.is_empty()
+ && let Err(error) = self
+ .mark_or_delete_processed_rows(
+ pool,
+ &batch.table,
+ pk_column,
+ &batch.processed_ids,
+ )
+ .await
+ {
+ error!(
+ "Failed to mark or delete processed rows for table '{}', skipping this cycle: {error}",
+ batch.table
+ );
+ continue;
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ state_updates.push((batch.table, offset));
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
+ }
+ state.last_poll_time = Utc::now();
+ }
+
+ Ok(messages)
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, Error> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.validate_custom_query(custom_query)?;
+ self.substitute_query_params(custom_query, table, last_offset, batch_size)
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ for row in rows {
+ let processed = self.process_row(&row, &table_config)?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+ if let Some(offset) = processed.max_offset {
+ batch.max_offset = Some(offset);
+ }
+
+ batch.messages.push(processed.message);
+ }
+
+ Ok(batch)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<String> = None;
+ let mut max_offset: Option<String> = None;
+ let mut extracted_payload: Option<Vec<u8>> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ if name == config.payload_col {
+ extracted_payload =
+ Some(self.extract_payload_column(row, i, config.payload_format)?);
+ }
+ if name == config.tracking_column {
+ max_offset = value_as_string(&extract_column_value(row, i)?);
+ }
+ if name == config.pk_column {
+ row_pk = value_as_string(&extract_column_value(row, i)?);
+ }
+ }
+ }
+
+ if extracted_payload.is_none() {
+ let mut data = serde_json::Map::new();
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ let column_name = if config.snake_case_columns {
+ to_snake_case(name)
+ } else {
+ name.to_string()
+ };
+ let value = extract_column_value(row, i)?;
+ if name == config.tracking_column {
+ max_offset = value_as_string(&value);
+ }
+ if name == config.pk_column {
+ row_pk = value_as_string(&value);
+ }
+ data.insert(column_name, value);
+ }
+
+ extracted_payload = Some(if config.include_metadata {
+ let record = DatabaseRecord {
+ table_name: config.table.to_string(),
+ operation_type: "SELECT".to_string(),
+ timestamp: Utc::now(),
+ data: serde_json::Value::Object(data),
+ old_data: None,
+ };
+ simd_json::to_vec(&record).map_err(|_| Error::InvalidRecord)?
+ } else {
+ simd_json::to_vec(&data).map_err(|_| Error::InvalidRecord)?
+ });
+ }
+
+ // Both paths above always assign extracted_payload before reaching here.
+ Ok(build_processed_row(
+ extracted_payload.unwrap(),
+ max_offset,
+ row_pk,
+ ))
+ }
+}
+
+fn extract_column_value(row: &MySqlRow, column_index: usize) -> Result<serde_json::Value, Error> {
+ let column = &row.columns()[column_index];
+ let type_name = column.type_info().name();
+
+ match type_name {
+ "BOOLEAN" => {
+ let value: Option<bool> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::Bool)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT" => {
+ let value: Option<i8> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT" => {
+ let value: Option<i16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT" | "INT" => {
+ let value: Option<i32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT" => {
+ let value: Option<i64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT UNSIGNED" => {
+ let value: Option<u8> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT UNSIGNED" => {
+ let value: Option<u16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT UNSIGNED" | "INT UNSIGNED" => {
+ let value: Option<u32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT UNSIGNED" => {
+ let value: Option<u64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "FLOAT" => {
+ let value: Option<f32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as f64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DOUBLE" => {
+ let value: Option<f64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DECIMAL" => {
+ let value: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIT" => {
+ let value: Option<u64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "YEAR" => {
+ let value: Option<u16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATE" => {
+ let value: Option<NaiveDate> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|d| serde_json::Value::String(d.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TIME" => {
+ let value: Option<NaiveTime> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|t| serde_json::Value::String(t.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATETIME" | "TIMESTAMP" => {
Review Comment:
same class of bug as the DECIMAL one: `NaiveDateTime` in sqlx-mysql does not override `compatible()`, so it accepts `DATETIME` only, never `TIMESTAMP` (only `DateTime<Utc>`/`DateTime<Local>` accept both). any table with a TIMESTAMP column (`created_at`/`updated_at` - the natural tracking columns) fails every row and the table stalls permanently.
fix: decode TIMESTAMP via `DateTime<Utc>` and keep `NaiveDateTime` for DATETIME. also worth a test with a TIMESTAMP tracking column.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).execute(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. A table's messages are emitted
+ // only after its rows are marked or deleted, so a failure here can never
+ // leave a table marked-but-unpublished. `mark_or_delete_processed_rows`
+ // retries transient errors; a permanent failure isolates that one table
+ // (its rows stay in MySQL and are retried next cycle) while the others
+ // still flow.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if !batch.processed_ids.is_empty()
Review Comment:
if the pk value can't be extracted (custom query not projecting it, misconfigured `primary_key_column`, or a non-scalar value - `value_as_string` returns `None` for anything but strings/numbers), `processed_ids` stays empty and the mark/delete is silently skipped while the messages still publish and the offset advances. with delete_after_read that leaks rows that look processed. worth a warn log when delete/mark is configured but no pks were extracted.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).execute(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. A table's messages are emitted
+ // only after its rows are marked or deleted, so a failure here can never
+ // leave a table marked-but-unpublished. `mark_or_delete_processed_rows`
+ // retries transient errors; a permanent failure isolates that one table
+ // (its rows stay in MySQL and are retried next cycle) while the others
+ // still flow.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if !batch.processed_ids.is_empty()
+ && let Err(error) = self
+ .mark_or_delete_processed_rows(
+ pool,
+ &batch.table,
+ pk_column,
+ &batch.processed_ids,
+ )
+ .await
+ {
+ error!(
+ "Failed to mark or delete processed rows for table '{}', skipping this cycle: {error}",
+ batch.table
+ );
+ continue;
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ state_updates.push((batch.table, offset));
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
+ }
+ state.last_poll_time = Utc::now();
+ }
+
+ Ok(messages)
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, Error> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.validate_custom_query(custom_query)?;
+ self.substitute_query_params(custom_query, table, last_offset, batch_size)
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ for row in rows {
+ let processed = self.process_row(&row, &table_config)?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+ if let Some(offset) = processed.max_offset {
+ batch.max_offset = Some(offset);
+ }
+
+ batch.messages.push(processed.message);
+ }
+
+ Ok(batch)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<String> = None;
+ let mut max_offset: Option<String> = None;
+ let mut extracted_payload: Option<Vec<u8>> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ if name == config.payload_col {
+ extracted_payload =
+ Some(self.extract_payload_column(row, i, config.payload_format)?);
+ }
+ if name == config.tracking_column {
+ max_offset = value_as_string(&extract_column_value(row, i)?);
+ }
+ if name == config.pk_column {
+ row_pk = value_as_string(&extract_column_value(row, i)?);
+ }
+ }
+ }
+
+ if extracted_payload.is_none() {
+ let mut data = serde_json::Map::new();
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ let column_name = if config.snake_case_columns {
+ to_snake_case(name)
+ } else {
+ name.to_string()
+ };
+ let value = extract_column_value(row, i)?;
+ if name == config.tracking_column {
+ max_offset = value_as_string(&value);
+ }
+ if name == config.pk_column {
+ row_pk = value_as_string(&value);
+ }
+ data.insert(column_name, value);
+ }
+
+ extracted_payload = Some(if config.include_metadata {
+ let record = DatabaseRecord {
+ table_name: config.table.to_string(),
+ operation_type: "SELECT".to_string(),
+ timestamp: Utc::now(),
+ data: serde_json::Value::Object(data),
+ old_data: None,
+ };
+ simd_json::to_vec(&record).map_err(|_| Error::InvalidRecord)?
+ } else {
+ simd_json::to_vec(&data).map_err(|_| Error::InvalidRecord)?
+ });
+ }
+
+ // Both paths above always assign extracted_payload before reaching here.
+ Ok(build_processed_row(
+ extracted_payload.unwrap(),
+ max_offset,
+ row_pk,
+ ))
+ }
+}
+
+fn extract_column_value(row: &MySqlRow, column_index: usize) -> Result<serde_json::Value, Error> {
+ let column = &row.columns()[column_index];
+ let type_name = column.type_info().name();
+
+ match type_name {
+ "BOOLEAN" => {
+ let value: Option<bool> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::Bool)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT" => {
+ let value: Option<i8> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT" => {
+ let value: Option<i16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT" | "INT" => {
+ let value: Option<i32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT" => {
+ let value: Option<i64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT UNSIGNED" => {
+ let value: Option<u8> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT UNSIGNED" => {
+ let value: Option<u16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT UNSIGNED" | "INT UNSIGNED" => {
+ let value: Option<u32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT UNSIGNED" => {
+ let value: Option<u64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "FLOAT" => {
+ let value: Option<f32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as f64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DOUBLE" => {
+ let value: Option<f64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DECIMAL" => {
+ let value: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIT" => {
+ let value: Option<u64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "YEAR" => {
+ let value: Option<u16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATE" => {
+ let value: Option<NaiveDate> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|d| serde_json::Value::String(d.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TIME" => {
+ let value: Option<NaiveTime> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|t| serde_json::Value::String(t.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATETIME" | "TIMESTAMP" => {
+ let value: Option<NaiveDateTime> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|dt| serde_json::Value::String(dt.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT" | "MEDIUMTEXT" | "LONGTEXT" | "ENUM" | "SET" => {
+ let value: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BINARY" | "VARBINARY" | "TINYBLOB" | "BLOB" | "MEDIUMBLOB" | "LONGBLOB" | "GEOMETRY" => {
+ let value: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|bytes| {
+ serde_json::Value::String(
+ base64::engine::general_purpose::STANDARD.encode(&bytes),
+ )
+ })
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "JSON" => {
+ let value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value.unwrap_or(serde_json::Value::Null))
+ }
+ "NULL" => Ok(serde_json::Value::Null),
+ _ => {
+ let column_name = column.name();
+ warn!(
+ "Column '{column_name}' has unrecognized MySQL type '{type_name}', \
+ attempting text extraction"
+ );
+ if let Ok(text) = row.try_get::<Option<String>, _>(column_index) {
+ return Ok(text
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null));
+ }
+ if let Ok(bytes) = row.try_get::<Option<Vec<u8>>, _>(column_index) {
+ return Ok(bytes
+ .map(|b| {
+ serde_json::Value::String(
+ base64::engine::general_purpose::STANDARD.encode(&b),
+ )
+ })
+ .unwrap_or(serde_json::Value::Null));
+ }
+ error!(
+ "Column '{column_name}' has unsupported MySQL type '{type_name}', \
+ returning null"
+ );
+ Ok(serde_json::Value::Null)
+ }
+ }
+}
+
+fn value_as_string(value: &serde_json::Value) -> Option<String> {
+ match value {
+ serde_json::Value::String(s) => Some(s.clone()),
+ serde_json::Value::Number(n) => Some(n.to_string()),
+ _ => None,
+ }
+}
+
+fn build_processed_row(
+ payload: Vec<u8>,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+) -> ProcessedRow {
+ let now = Utc::now().timestamp_millis() as u64;
+ ProcessedRow {
+ message: ProducedMessage {
+ id: Some(Uuid::new_v4().as_u128()),
Review Comment:
message id is a random uuid per row, so replayed rows after a restart get new ids and downstream can't dedup them - which undercuts the offset-tracking/duplicates story in the readme. a deterministic id (hash of table + pk, say) makes replays idempotent and drops the per-row rng call.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).execute(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. A table's messages are emitted
+ // only after its rows are marked or deleted, so a failure here can never
+ // leave a table marked-but-unpublished. `mark_or_delete_processed_rows`
+ // retries transient errors; a permanent failure isolates that one table
+ // (its rows stay in MySQL and are retried next cycle) while the others
+ // still flow.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if !batch.processed_ids.is_empty()
+ && let Err(error) = self
+ .mark_or_delete_processed_rows(
+ pool,
+ &batch.table,
+ pk_column,
+ &batch.processed_ids,
+ )
+ .await
+ {
+ error!(
+ "Failed to mark or delete processed rows for table '{}', skipping this cycle: {error}",
+ batch.table
+ );
+ continue;
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ state_updates.push((batch.table, offset));
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
+ }
+ state.last_poll_time = Utc::now();
+ }
+
+ Ok(messages)
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, Error> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.validate_custom_query(custom_query)?;
+ self.substitute_query_params(custom_query, table, last_offset, batch_size)
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ for row in rows {
+ let processed = self.process_row(&row, &table_config)?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+ if let Some(offset) = processed.max_offset {
+ batch.max_offset = Some(offset);
Review Comment:
for custom queries this ends up being the last returned row's tracking value, but nothing enforces `ORDER BY <tracking> ASC` on that path (the built query adds it at line 399). an unordered query with LIMIT saves a cursor below the true max - rows get skipped or re-emitted on the next `> $offset` poll. either inject the order-by for custom queries or make the readme require it.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).execute(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. A table's messages are emitted
+ // only after its rows are marked or deleted, so a failure here can never
+ // leave a table marked-but-unpublished. `mark_or_delete_processed_rows`
+ // retries transient errors; a permanent failure isolates that one table
+ // (its rows stay in MySQL and are retried next cycle) while the others
+ // still flow.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if !batch.processed_ids.is_empty()
+ && let Err(error) = self
+ .mark_or_delete_processed_rows(
+ pool,
+ &batch.table,
+ pk_column,
+ &batch.processed_ids,
+ )
+ .await
+ {
+ error!(
+ "Failed to mark or delete processed rows for table '{}', skipping this cycle: {error}",
+ batch.table
+ );
+ continue;
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ state_updates.push((batch.table, offset));
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
+ }
+ state.last_poll_time = Utc::now();
Review Comment:
`last_poll_time` changes every cycle and poll always returns `Some(state)`, so the runtime rewrites and fsyncs the state file every poll even when nothing was fetched. the field is also never read anywhere. dropping it and returning `None` when offsets/processed_rows are unchanged makes idle cycles free. safe to change the state shape - new connector, and invalid persisted state already falls back to a fresh start.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
Review Comment:
the numeric check only tries `i64`, so a `BIGINT UNSIGNED` pk above `i64::MAX` gets quoted as a string and mysql compares it as a float in the IN list - precision loss can match the wrong row or none. `format_offset_value` accepts i64 or f64, so the two paths disagree; add a `u64` parse here.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})")
+ } else if let Some(processed_col) = &self.config.processed_column {
Review Comment:
when both `delete_after_read` and `processed_column` are set, delete silently wins and the processed column only survives as a query filter. worth a validation error or at least a warn in `open()`.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).execute(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. A table's messages are emitted
+ // only after its rows are marked or deleted, so a failure here can never
+ // leave a table marked-but-unpublished. `mark_or_delete_processed_rows`
+ // retries transient errors; a permanent failure isolates that one table
+ // (its rows stay in MySQL and are retried next cycle) while the others
+ // still flow.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if !batch.processed_ids.is_empty()
+ && let Err(error) = self
+ .mark_or_delete_processed_rows(
+ pool,
+ &batch.table,
+ pk_column,
+ &batch.processed_ids,
+ )
+ .await
+ {
+ error!(
+ "Failed to mark or delete processed rows for table '{}', skipping this cycle: {error}",
+ batch.table
+ );
+ continue;
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ state_updates.push((batch.table, offset));
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
+ }
+ state.last_poll_time = Utc::now();
+ }
+
+ Ok(messages)
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, Error> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.validate_custom_query(custom_query)?;
+ self.substitute_query_params(custom_query, table, last_offset, batch_size)
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ for row in rows {
+ let processed = self.process_row(&row, &table_config)?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+ if let Some(offset) = processed.max_offset {
+ batch.max_offset = Some(offset);
+ }
+
+ batch.messages.push(processed.message);
+ }
+
+ Ok(batch)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<String> = None;
+ let mut max_offset: Option<String> = None;
+ let mut extracted_payload: Option<Vec<u8>> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ if name == config.payload_col {
+ extracted_payload =
+ Some(self.extract_payload_column(row, i, config.payload_format)?);
+ }
+ if name == config.tracking_column {
+ max_offset = value_as_string(&extract_column_value(row, i)?);
+ }
+ if name == config.pk_column {
+ row_pk = value_as_string(&extract_column_value(row, i)?);
+ }
+ }
+ }
+
+ if extracted_payload.is_none() {
+ let mut data = serde_json::Map::new();
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ let column_name = if config.snake_case_columns {
+ to_snake_case(name)
+ } else {
+ name.to_string()
+ };
+ let value = extract_column_value(row, i)?;
+ if name == config.tracking_column {
+ max_offset = value_as_string(&value);
+ }
+ if name == config.pk_column {
+ row_pk = value_as_string(&value);
+ }
+ data.insert(column_name, value);
+ }
+
+ extracted_payload = Some(if config.include_metadata {
+ let record = DatabaseRecord {
+ table_name: config.table.to_string(),
+ operation_type: "SELECT".to_string(),
+ timestamp: Utc::now(),
+ data: serde_json::Value::Object(data),
+ old_data: None,
+ };
+ simd_json::to_vec(&record).map_err(|_| Error::InvalidRecord)?
+ } else {
+ simd_json::to_vec(&data).map_err(|_| Error::InvalidRecord)?
+ });
+ }
+
+ // Both paths above always assign extracted_payload before reaching here.
+ Ok(build_processed_row(
+ extracted_payload.unwrap(),
+ max_offset,
+ row_pk,
+ ))
+ }
+}
+
+fn extract_column_value(row: &MySqlRow, column_index: usize) -> Result<serde_json::Value, Error> {
+ let column = &row.columns()[column_index];
+ let type_name = column.type_info().name();
+
+ match type_name {
+ "BOOLEAN" => {
+ let value: Option<bool> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::Bool)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT" => {
+ let value: Option<i8> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT" => {
+ let value: Option<i16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT" | "INT" => {
+ let value: Option<i32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT" => {
+ let value: Option<i64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT UNSIGNED" => {
+ let value: Option<u8> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT UNSIGNED" => {
+ let value: Option<u16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT UNSIGNED" | "INT UNSIGNED" => {
+ let value: Option<u32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT UNSIGNED" => {
+ let value: Option<u64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "FLOAT" => {
+ let value: Option<f32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as f64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DOUBLE" => {
+ let value: Option<f64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DECIMAL" => {
+ let value: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIT" => {
+ let value: Option<u64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "YEAR" => {
+ let value: Option<u16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATE" => {
+ let value: Option<NaiveDate> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|d| serde_json::Value::String(d.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TIME" => {
+ let value: Option<NaiveTime> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|t| serde_json::Value::String(t.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATETIME" | "TIMESTAMP" => {
+ let value: Option<NaiveDateTime> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|dt| serde_json::Value::String(dt.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT" | "MEDIUMTEXT" | "LONGTEXT" | "ENUM" | "SET" => {
+ let value: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BINARY" | "VARBINARY" | "TINYBLOB" | "BLOB" | "MEDIUMBLOB" | "LONGBLOB" | "GEOMETRY" => {
+ let value: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|bytes| {
+ serde_json::Value::String(
+ base64::engine::general_purpose::STANDARD.encode(&bytes),
+ )
+ })
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "JSON" => {
+ let value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value.unwrap_or(serde_json::Value::Null))
+ }
+ "NULL" => Ok(serde_json::Value::Null),
+ _ => {
+ let column_name = column.name();
+ warn!(
+ "Column '{column_name}' has unrecognized MySQL type '{type_name}', \
+ attempting text extraction"
+ );
+ if let Ok(text) = row.try_get::<Option<String>, _>(column_index) {
+ return Ok(text
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null));
+ }
+ if let Ok(bytes) = row.try_get::<Option<Vec<u8>>, _>(column_index) {
+ return Ok(bytes
+ .map(|b| {
+ serde_json::Value::String(
+ base64::engine::general_purpose::STANDARD.encode(&b),
+ )
+ })
+ .unwrap_or(serde_json::Value::Null));
+ }
+ error!(
+ "Column '{column_name}' has unsupported MySQL type '{type_name}', \
+ returning null"
+ );
+ Ok(serde_json::Value::Null)
+ }
+ }
+}
+
+fn value_as_string(value: &serde_json::Value) -> Option<String> {
+ match value {
+ serde_json::Value::String(s) => Some(s.clone()),
+ serde_json::Value::Number(n) => Some(n.to_string()),
+ _ => None,
+ }
+}
+
+fn build_processed_row(
+ payload: Vec<u8>,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+) -> ProcessedRow {
+ let now = Utc::now().timestamp_millis() as u64;
+ ProcessedRow {
+ message: ProducedMessage {
+ id: Some(Uuid::new_v4().as_u128()),
+ headers: None,
+ checksum: None,
+ timestamp: Some(now),
+ origin_timestamp: Some(now),
+ payload,
+ },
+ max_offset,
+ row_pk,
+ }
+}
+
+fn to_snake_case(input: &str) -> String {
+ let mut result = String::new();
+ let mut prev_was_uppercase = false;
+ for (i, ch) in input.chars().enumerate() {
+ if ch.is_uppercase() {
+ if i > 0 && !prev_was_uppercase {
+ result.push('_');
+ }
+ if let Some(lc) = ch.to_lowercase().next() {
+ result.push(lc);
+ } else {
+ result.push(ch);
+ }
+ prev_was_uppercase = true;
+ } else {
+ result.push(ch);
+ prev_was_uppercase = false;
+ }
+ }
+ result
+}
+
+fn redact_connection_string(conn_str: &str) -> String {
+ if let Some(scheme_end) = conn_str.find("://") {
+ let scheme = &conn_str[..scheme_end + 3];
+ let rest = &conn_str[scheme_end + 3..];
+ let preview: String = rest.chars().take(3).collect();
+ return format!("{scheme}{preview}***");
+ }
+ let preview: String = conn_str.chars().take(3).collect();
+ format!("{preview}***")
+}
+
+fn quote_identifier(name: &str) -> Result<String, Error> {
+ if name.is_empty() {
+ return Err(Error::InvalidConfigValue(
+ "identifier must not be empty".to_string(),
+ ));
+ }
+ if name.contains('\0') {
+ return Err(Error::InvalidConfigValue(format!(
+ "identifier '{name}' contains NUL byte"
+ )));
+ }
+ let escaped = name.replace('`', "``");
+ Ok(format!("`{escaped}`"))
+}
+
+fn quote_qualified_identifier(name: &str) -> Result<String, Error> {
+ if !name.contains('.') {
+ return quote_identifier(name);
+ }
+ let parts: Result<Vec<_>, _> = name.split('.').map(quote_identifier).collect();
+ Ok(parts?.join("."))
+}
+
+fn format_offset_value(value: &str) -> String {
+ if value.parse::<i64>().is_ok() || value.parse::<f64>().is_ok_and(|v| v.is_finite()) {
+ value.to_string()
+ } else {
+ let escaped = value
+ .replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "");
+ format!("'{escaped}'")
+ }
+}
+
+fn is_transient_error(e: &sqlx::Error) -> bool {
+ match e {
+ sqlx::Error::Io(_) => true,
+ sqlx::Error::PoolTimedOut => true,
+ sqlx::Error::PoolClosed => false,
+ sqlx::Error::Protocol(_) => false,
+ // MySQL surfaces a numeric error code (e.g. 1213) and a SQLSTATE (e.g. "40001").
+ // `DatabaseError::code()` returns the SQLSTATE, so matching it against MySQL error
+ // numbers never fires. Downcast to the driver error and compare `number()` instead.
+ sqlx::Error::Database(db_err) => db_err
+ .try_downcast_ref::<MySqlDatabaseError>()
+ .is_some_and(|mysql_err| {
+ matches!(
+ mysql_err.number(),
+ // concurrency
+ 1213 | 1205 |
+ // server unavailability
+ 1053 | 1152 | 1080 |
+ // connection/network
+ 2006 | 2013 | 1158 | 1159 | 1160 | 1161 |
+ // resource exhaustion
+ 1040 | 1041
+ )
+ }),
+ _ => false,
+ }
+}
+
+async fn with_retry<T, F, Fut>(operation: F, max_retries: u32, delay_ms: u64) -> Result<T, Error>
+where
+ F: Fn() -> Fut,
+ Fut: std::future::Future<Output = Result<T, sqlx::Error>>,
+{
+ let mut attempts = 0;
+ loop {
+ match operation().await {
+ Ok(result) => return Ok(result),
+ Err(e) => {
+ attempts += 1;
+ if attempts >= max_retries || !is_transient_error(&e) {
+ error!("Database operation failed after {attempts} attempts: {e}");
+ return Err(Error::InvalidRecord);
Review Comment:
every db failure - connection loss, pool timeout, auth - surfaces as `Error::InvalidRecord`, so a dead database reads as a data problem in the upstream logs ("skipping this cycle: Invalid record"). the sdk has more accurate variants for this.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).execute(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. A table's messages are emitted
+ // only after its rows are marked or deleted, so a failure here can never
+ // leave a table marked-but-unpublished. `mark_or_delete_processed_rows`
+ // retries transient errors; a permanent failure isolates that one table
+ // (its rows stay in MySQL and are retried next cycle) while the others
+ // still flow.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if !batch.processed_ids.is_empty()
+ && let Err(error) = self
+ .mark_or_delete_processed_rows(
+ pool,
+ &batch.table,
+ pk_column,
+ &batch.processed_ids,
+ )
+ .await
+ {
+ error!(
+ "Failed to mark or delete processed rows for table '{}', skipping this cycle: {error}",
+ batch.table
+ );
+ continue;
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ state_updates.push((batch.table, offset));
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
+ }
+ state.last_poll_time = Utc::now();
+ }
+
+ Ok(messages)
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, Error> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.validate_custom_query(custom_query)?;
+ self.substitute_query_params(custom_query, table, last_offset, batch_size)
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ for row in rows {
+ let processed = self.process_row(&row, &table_config)?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+ if let Some(offset) = processed.max_offset {
+ batch.max_offset = Some(offset);
+ }
+
+ batch.messages.push(processed.message);
+ }
+
+ Ok(batch)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<String> = None;
+ let mut max_offset: Option<String> = None;
+ let mut extracted_payload: Option<Vec<u8>> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ if name == config.payload_col {
+ extracted_payload =
+ Some(self.extract_payload_column(row, i, config.payload_format)?);
+ }
+ if name == config.tracking_column {
+ max_offset = value_as_string(&extract_column_value(row, i)?);
+ }
+ if name == config.pk_column {
+ row_pk = value_as_string(&extract_column_value(row, i)?);
+ }
+ }
+ }
+
+ if extracted_payload.is_none() {
+ let mut data = serde_json::Map::new();
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ let column_name = if config.snake_case_columns {
+ to_snake_case(name)
+ } else {
+ name.to_string()
+ };
+ let value = extract_column_value(row, i)?;
+ if name == config.tracking_column {
+ max_offset = value_as_string(&value);
+ }
+ if name == config.pk_column {
+ row_pk = value_as_string(&value);
+ }
+ data.insert(column_name, value);
+ }
+
+ extracted_payload = Some(if config.include_metadata {
+ let record = DatabaseRecord {
+ table_name: config.table.to_string(),
+ operation_type: "SELECT".to_string(),
+ timestamp: Utc::now(),
+ data: serde_json::Value::Object(data),
+ old_data: None,
+ };
+ simd_json::to_vec(&record).map_err(|_| Error::InvalidRecord)?
+ } else {
+ simd_json::to_vec(&data).map_err(|_| Error::InvalidRecord)?
+ });
+ }
+
+ // Both paths above always assign extracted_payload before reaching here.
+ Ok(build_processed_row(
+ extracted_payload.unwrap(),
+ max_offset,
+ row_pk,
+ ))
+ }
+}
+
+fn extract_column_value(row: &MySqlRow, column_index: usize) -> Result<serde_json::Value, Error> {
+ let column = &row.columns()[column_index];
+ let type_name = column.type_info().name();
+
+ match type_name {
+ "BOOLEAN" => {
+ let value: Option<bool> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::Bool)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT" => {
+ let value: Option<i8> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT" => {
+ let value: Option<i16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT" | "INT" => {
+ let value: Option<i32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT" => {
+ let value: Option<i64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT UNSIGNED" => {
+ let value: Option<u8> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT UNSIGNED" => {
+ let value: Option<u16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT UNSIGNED" | "INT UNSIGNED" => {
+ let value: Option<u32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT UNSIGNED" => {
+ let value: Option<u64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "FLOAT" => {
+ let value: Option<f32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as f64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DOUBLE" => {
+ let value: Option<f64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DECIMAL" => {
+ let value: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIT" => {
+ let value: Option<u64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "YEAR" => {
+ let value: Option<u16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATE" => {
+ let value: Option<NaiveDate> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|d| serde_json::Value::String(d.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TIME" => {
+ let value: Option<NaiveTime> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|t| serde_json::Value::String(t.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATETIME" | "TIMESTAMP" => {
+ let value: Option<NaiveDateTime> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|dt| serde_json::Value::String(dt.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT" | "MEDIUMTEXT" | "LONGTEXT" | "ENUM" | "SET" => {
+ let value: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BINARY" | "VARBINARY" | "TINYBLOB" | "BLOB" | "MEDIUMBLOB" | "LONGBLOB" | "GEOMETRY" => {
Review Comment:
`GEOMETRY` hits the same sqlx compatibility gate as DECIMAL/TIMESTAMP - the byte-slice impl excludes geometry, so this arm fails on every row instead of base64-encoding. rare type, but the readme lists it as supported; either handle it in the same pass as the other two or drop it from the readme.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).execute(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. A table's messages are emitted
+ // only after its rows are marked or deleted, so a failure here can never
+ // leave a table marked-but-unpublished. `mark_or_delete_processed_rows`
+ // retries transient errors; a permanent failure isolates that one table
+ // (its rows stay in MySQL and are retried next cycle) while the others
+ // still flow.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if !batch.processed_ids.is_empty()
+ && let Err(error) = self
+ .mark_or_delete_processed_rows(
+ pool,
+ &batch.table,
+ pk_column,
+ &batch.processed_ids,
+ )
+ .await
+ {
+ error!(
+ "Failed to mark or delete processed rows for table '{}', skipping this cycle: {error}",
+ batch.table
+ );
+ continue;
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ state_updates.push((batch.table, offset));
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
+ }
+ state.last_poll_time = Utc::now();
+ }
+
+ Ok(messages)
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, Error> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.validate_custom_query(custom_query)?;
+ self.substitute_query_params(custom_query, table, last_offset, batch_size)
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ for row in rows {
+ let processed = self.process_row(&row, &table_config)?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+ if let Some(offset) = processed.max_offset {
+ batch.max_offset = Some(offset);
+ }
+
+ batch.messages.push(processed.message);
+ }
+
+ Ok(batch)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<String> = None;
+ let mut max_offset: Option<String> = None;
+ let mut extracted_payload: Option<Vec<u8>> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ if name == config.payload_col {
+ extracted_payload =
+ Some(self.extract_payload_column(row, i, config.payload_format)?);
+ }
+ if name == config.tracking_column {
+ max_offset = value_as_string(&extract_column_value(row, i)?);
+ }
+ if name == config.pk_column {
+ row_pk = value_as_string(&extract_column_value(row, i)?);
+ }
+ }
+ }
+
+ if extracted_payload.is_none() {
+ let mut data = serde_json::Map::new();
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ let column_name = if config.snake_case_columns {
+ to_snake_case(name)
+ } else {
+ name.to_string()
+ };
+ let value = extract_column_value(row, i)?;
+ if name == config.tracking_column {
+ max_offset = value_as_string(&value);
+ }
+ if name == config.pk_column {
+ row_pk = value_as_string(&value);
+ }
+ data.insert(column_name, value);
+ }
+
+ extracted_payload = Some(if config.include_metadata {
+ let record = DatabaseRecord {
+ table_name: config.table.to_string(),
+ operation_type: "SELECT".to_string(),
+ timestamp: Utc::now(),
+ data: serde_json::Value::Object(data),
+ old_data: None,
+ };
+ simd_json::to_vec(&record).map_err(|_| Error::InvalidRecord)?
+ } else {
+ simd_json::to_vec(&data).map_err(|_| Error::InvalidRecord)?
+ });
+ }
+
+ // Both paths above always assign extracted_payload before reaching here.
+ Ok(build_processed_row(
+ extracted_payload.unwrap(),
+ max_offset,
+ row_pk,
+ ))
+ }
+}
+
+fn extract_column_value(row: &MySqlRow, column_index: usize) -> Result<serde_json::Value, Error> {
+ let column = &row.columns()[column_index];
+ let type_name = column.type_info().name();
+
+ match type_name {
+ "BOOLEAN" => {
+ let value: Option<bool> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::Bool)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT" => {
+ let value: Option<i8> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT" => {
+ let value: Option<i16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT" | "INT" => {
+ let value: Option<i32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT" => {
+ let value: Option<i64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT UNSIGNED" => {
+ let value: Option<u8> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT UNSIGNED" => {
+ let value: Option<u16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT UNSIGNED" | "INT UNSIGNED" => {
+ let value: Option<u32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT UNSIGNED" => {
+ let value: Option<u64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "FLOAT" => {
+ let value: Option<f32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as f64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DOUBLE" => {
+ let value: Option<f64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DECIMAL" => {
+ let value: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIT" => {
+ let value: Option<u64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "YEAR" => {
+ let value: Option<u16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATE" => {
+ let value: Option<NaiveDate> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|d| serde_json::Value::String(d.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TIME" => {
+ let value: Option<NaiveTime> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|t| serde_json::Value::String(t.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATETIME" | "TIMESTAMP" => {
+ let value: Option<NaiveDateTime> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|dt| serde_json::Value::String(dt.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT" | "MEDIUMTEXT" | "LONGTEXT" | "ENUM" | "SET" => {
+ let value: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BINARY" | "VARBINARY" | "TINYBLOB" | "BLOB" | "MEDIUMBLOB" | "LONGBLOB" | "GEOMETRY" => {
+ let value: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|bytes| {
+ serde_json::Value::String(
+ base64::engine::general_purpose::STANDARD.encode(&bytes),
+ )
+ })
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "JSON" => {
+ let value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value.unwrap_or(serde_json::Value::Null))
+ }
+ "NULL" => Ok(serde_json::Value::Null),
+ _ => {
+ let column_name = column.name();
+ warn!(
+ "Column '{column_name}' has unrecognized MySQL type '{type_name}', \
+ attempting text extraction"
+ );
+ if let Ok(text) = row.try_get::<Option<String>, _>(column_index) {
+ return Ok(text
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null));
+ }
+ if let Ok(bytes) = row.try_get::<Option<Vec<u8>>, _>(column_index) {
+ return Ok(bytes
+ .map(|b| {
+ serde_json::Value::String(
+ base64::engine::general_purpose::STANDARD.encode(&b),
+ )
+ })
+ .unwrap_or(serde_json::Value::Null));
+ }
+ error!(
+ "Column '{column_name}' has unsupported MySQL type '{type_name}', \
+ returning null"
+ );
+ Ok(serde_json::Value::Null)
+ }
+ }
+}
+
+fn value_as_string(value: &serde_json::Value) -> Option<String> {
+ match value {
+ serde_json::Value::String(s) => Some(s.clone()),
+ serde_json::Value::Number(n) => Some(n.to_string()),
+ _ => None,
+ }
+}
+
+fn build_processed_row(
+ payload: Vec<u8>,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+) -> ProcessedRow {
+ let now = Utc::now().timestamp_millis() as u64;
+ ProcessedRow {
+ message: ProducedMessage {
+ id: Some(Uuid::new_v4().as_u128()),
+ headers: None,
+ checksum: None,
+ timestamp: Some(now),
+ origin_timestamp: Some(now),
+ payload,
+ },
+ max_offset,
+ row_pk,
+ }
+}
+
+fn to_snake_case(input: &str) -> String {
+ let mut result = String::new();
+ let mut prev_was_uppercase = false;
+ for (i, ch) in input.chars().enumerate() {
+ if ch.is_uppercase() {
+ if i > 0 && !prev_was_uppercase {
+ result.push('_');
+ }
+ if let Some(lc) = ch.to_lowercase().next() {
+ result.push(lc);
+ } else {
+ result.push(ch);
+ }
+ prev_was_uppercase = true;
+ } else {
+ result.push(ch);
+ prev_was_uppercase = false;
+ }
+ }
+ result
+}
+
+fn redact_connection_string(conn_str: &str) -> String {
+ if let Some(scheme_end) = conn_str.find("://") {
+ let scheme = &conn_str[..scheme_end + 3];
+ let rest = &conn_str[scheme_end + 3..];
+ let preview: String = rest.chars().take(3).collect();
+ return format!("{scheme}{preview}***");
+ }
+ let preview: String = conn_str.chars().take(3).collect();
+ format!("{preview}***")
+}
+
+fn quote_identifier(name: &str) -> Result<String, Error> {
+ if name.is_empty() {
+ return Err(Error::InvalidConfigValue(
+ "identifier must not be empty".to_string(),
+ ));
+ }
+ if name.contains('\0') {
+ return Err(Error::InvalidConfigValue(format!(
+ "identifier '{name}' contains NUL byte"
+ )));
+ }
+ let escaped = name.replace('`', "``");
+ Ok(format!("`{escaped}`"))
+}
+
+fn quote_qualified_identifier(name: &str) -> Result<String, Error> {
+ if !name.contains('.') {
+ return quote_identifier(name);
+ }
+ let parts: Result<Vec<_>, _> = name.split('.').map(quote_identifier).collect();
+ Ok(parts?.join("."))
+}
+
+fn format_offset_value(value: &str) -> String {
+ if value.parse::<i64>().is_ok() || value.parse::<f64>().is_ok_and(|v| v.is_finite()) {
+ value.to_string()
+ } else {
+ let escaped = value
+ .replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "");
+ format!("'{escaped}'")
+ }
+}
+
+fn is_transient_error(e: &sqlx::Error) -> bool {
+ match e {
+ sqlx::Error::Io(_) => true,
+ sqlx::Error::PoolTimedOut => true,
+ sqlx::Error::PoolClosed => false,
+ sqlx::Error::Protocol(_) => false,
+ // MySQL surfaces a numeric error code (e.g. 1213) and a SQLSTATE (e.g. "40001").
+ // `DatabaseError::code()` returns the SQLSTATE, so matching it against MySQL error
+ // numbers never fires. Downcast to the driver error and compare `number()` instead.
+ sqlx::Error::Database(db_err) => db_err
+ .try_downcast_ref::<MySqlDatabaseError>()
+ .is_some_and(|mysql_err| {
+ matches!(
+ mysql_err.number(),
+ // concurrency
+ 1213 | 1205 |
+ // server unavailability
+ 1053 | 1152 | 1080 |
+ // connection/network
+ 2006 | 2013 | 1158 | 1159 | 1160 | 1161 |
+ // resource exhaustion
+ 1040 | 1041
+ )
+ }),
+ _ => false,
+ }
+}
+
+async fn with_retry<T, F, Fut>(operation: F, max_retries: u32, delay_ms: u64) -> Result<T, Error>
+where
+ F: Fn() -> Fut,
+ Fut: std::future::Future<Output = Result<T, sqlx::Error>>,
+{
+ let mut attempts = 0;
+ loop {
+ match operation().await {
+ Ok(result) => return Ok(result),
+ Err(e) => {
+ attempts += 1;
+ if attempts >= max_retries || !is_transient_error(&e) {
Review Comment:
`attempts >= max_retries` makes `max_retries` actually the total attempt count: default 3 = one initial try + two retries, and `max_retries = 1` means zero retries. either the name/docs or the condition should agree.
##########
core/integration/tests/connectors/mysql/mod.rs:
##########
@@ -0,0 +1,32 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+mod mysql_source;
+
+use crate::connectors::TestMessage;
+use serde::Deserialize;
+
+const TEST_MESSAGE_COUNT: usize = 3;
Review Comment:
these consts + `DatabaseRecord` are byte-identical to the postgres test mod - could live next to `TestMessage` in the shared connectors mod. optional.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).execute(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. A table's messages are emitted
+ // only after its rows are marked or deleted, so a failure here can never
+ // leave a table marked-but-unpublished. `mark_or_delete_processed_rows`
+ // retries transient errors; a permanent failure isolates that one table
+ // (its rows stay in MySQL and are retried next cycle) while the others
+ // still flow.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if !batch.processed_ids.is_empty()
+ && let Err(error) = self
+ .mark_or_delete_processed_rows(
+ pool,
+ &batch.table,
+ pk_column,
+ &batch.processed_ids,
+ )
+ .await
+ {
+ error!(
+ "Failed to mark or delete processed rows for table '{}', skipping this cycle: {error}",
+ batch.table
+ );
+ continue;
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ state_updates.push((batch.table, offset));
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
+ }
+ state.last_poll_time = Utc::now();
+ }
+
+ Ok(messages)
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, Error> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.validate_custom_query(custom_query)?;
+ self.substitute_query_params(custom_query, table, last_offset, batch_size)
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ for row in rows {
+ let processed = self.process_row(&row, &table_config)?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+ if let Some(offset) = processed.max_offset {
+ batch.max_offset = Some(offset);
+ }
+
+ batch.messages.push(processed.message);
+ }
+
+ Ok(batch)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<String> = None;
+ let mut max_offset: Option<String> = None;
+ let mut extracted_payload: Option<Vec<u8>> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ if name == config.payload_col {
+ extracted_payload =
+ Some(self.extract_payload_column(row, i, config.payload_format)?);
+ }
+ if name == config.tracking_column {
+ max_offset = value_as_string(&extract_column_value(row, i)?);
Review Comment:
with the default config `tracking_column == pk_column`, so this loop decodes the same column twice through `extract_column_value` (and the non-payload path at 674-679 clones the value twice through `value_as_string`). extract once and reuse when the names match.
##########
core/connectors/sources/mysql_source/Cargo.toml:
##########
@@ -0,0 +1,53 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[package]
+name = "iggy_connector_mysql_source"
+version = "0.1.0"
+description = "Iggy MySQL source connector supporting table polling for message streaming platform"
+edition = "2024"
+license = "Apache-2.0"
+keywords = ["iggy", "messaging", "streaming", "mysql", "polling"]
+categories = ["command-line-utilities", "database", "network-programming"]
+homepage = "https://iggy.apache.org"
+documentation = "https://iggy.apache.org/docs"
+repository = "https://github.com/apache/iggy"
+readme = "README.md"
+publish = false
+
+[package.metadata.cargo-machete]
+ignored = ["dashmap", "simd-json"]
Review Comment:
the `simd-json` entry is redundant - machete detects the fully qualified `simd_json::` calls (random_source uses it the same way with no ignore and passes). `dashmap` genuinely needs the ignore since it's only used inside the `source_connector!` expansion. suggest `ignored = ["dashmap"]`.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).execute(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. A table's messages are emitted
+ // only after its rows are marked or deleted, so a failure here can never
+ // leave a table marked-but-unpublished. `mark_or_delete_processed_rows`
+ // retries transient errors; a permanent failure isolates that one table
+ // (its rows stay in MySQL and are retried next cycle) while the others
+ // still flow.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if !batch.processed_ids.is_empty()
+ && let Err(error) = self
+ .mark_or_delete_processed_rows(
+ pool,
+ &batch.table,
+ pk_column,
+ &batch.processed_ids,
+ )
+ .await
+ {
+ error!(
+ "Failed to mark or delete processed rows for table '{}', skipping this cycle: {error}",
+ batch.table
+ );
+ continue;
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ state_updates.push((batch.table, offset));
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
+ }
+ state.last_poll_time = Utc::now();
+ }
+
+ Ok(messages)
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, Error> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.validate_custom_query(custom_query)?;
+ self.substitute_query_params(custom_query, table, last_offset, batch_size)
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ for row in rows {
+ let processed = self.process_row(&row, &table_config)?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+ if let Some(offset) = processed.max_offset {
+ batch.max_offset = Some(offset);
+ }
+
+ batch.messages.push(processed.message);
+ }
+
+ Ok(batch)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<String> = None;
+ let mut max_offset: Option<String> = None;
+ let mut extracted_payload: Option<Vec<u8>> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ if name == config.payload_col {
+ extracted_payload =
+ Some(self.extract_payload_column(row, i, config.payload_format)?);
+ }
+ if name == config.tracking_column {
+ max_offset = value_as_string(&extract_column_value(row, i)?);
+ }
+ if name == config.pk_column {
+ row_pk = value_as_string(&extract_column_value(row, i)?);
+ }
+ }
+ }
+
+ if extracted_payload.is_none() {
+ let mut data = serde_json::Map::new();
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ let column_name = if config.snake_case_columns {
+ to_snake_case(name)
+ } else {
+ name.to_string()
+ };
+ let value = extract_column_value(row, i)?;
+ if name == config.tracking_column {
+ max_offset = value_as_string(&value);
+ }
+ if name == config.pk_column {
+ row_pk = value_as_string(&value);
+ }
+ data.insert(column_name, value);
+ }
+
+ extracted_payload = Some(if config.include_metadata {
+ let record = DatabaseRecord {
+ table_name: config.table.to_string(),
+ operation_type: "SELECT".to_string(),
+ timestamp: Utc::now(),
+ data: serde_json::Value::Object(data),
+ old_data: None,
+ };
+ simd_json::to_vec(&record).map_err(|_| Error::InvalidRecord)?
+ } else {
+ simd_json::to_vec(&data).map_err(|_| Error::InvalidRecord)?
+ });
+ }
+
+ // Both paths above always assign extracted_payload before reaching here.
+ Ok(build_processed_row(
+ extracted_payload.unwrap(),
+ max_offset,
+ row_pk,
+ ))
+ }
+}
+
+fn extract_column_value(row: &MySqlRow, column_index: usize) -> Result<serde_json::Value, Error> {
+ let column = &row.columns()[column_index];
+ let type_name = column.type_info().name();
+
+ match type_name {
+ "BOOLEAN" => {
+ let value: Option<bool> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::Bool)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT" => {
+ let value: Option<i8> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT" => {
+ let value: Option<i16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT" | "INT" => {
+ let value: Option<i32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT" => {
+ let value: Option<i64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT UNSIGNED" => {
+ let value: Option<u8> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT UNSIGNED" => {
+ let value: Option<u16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT UNSIGNED" | "INT UNSIGNED" => {
+ let value: Option<u32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT UNSIGNED" => {
+ let value: Option<u64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "FLOAT" => {
+ let value: Option<f32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as f64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DOUBLE" => {
+ let value: Option<f64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DECIMAL" => {
+ let value: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIT" => {
+ let value: Option<u64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "YEAR" => {
+ let value: Option<u16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATE" => {
+ let value: Option<NaiveDate> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|d| serde_json::Value::String(d.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TIME" => {
+ let value: Option<NaiveTime> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|t| serde_json::Value::String(t.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATETIME" | "TIMESTAMP" => {
+ let value: Option<NaiveDateTime> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|dt| serde_json::Value::String(dt.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT" | "MEDIUMTEXT" | "LONGTEXT" | "ENUM" | "SET" => {
+ let value: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BINARY" | "VARBINARY" | "TINYBLOB" | "BLOB" | "MEDIUMBLOB" | "LONGBLOB" | "GEOMETRY" => {
+ let value: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|bytes| {
+ serde_json::Value::String(
+ base64::engine::general_purpose::STANDARD.encode(&bytes),
+ )
+ })
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "JSON" => {
+ let value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value.unwrap_or(serde_json::Value::Null))
+ }
+ "NULL" => Ok(serde_json::Value::Null),
+ _ => {
+ let column_name = column.name();
+ warn!(
+ "Column '{column_name}' has unrecognized MySQL type '{type_name}', \
+ attempting text extraction"
+ );
+ if let Ok(text) = row.try_get::<Option<String>, _>(column_index) {
+ return Ok(text
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null));
+ }
+ if let Ok(bytes) = row.try_get::<Option<Vec<u8>>, _>(column_index) {
+ return Ok(bytes
+ .map(|b| {
+ serde_json::Value::String(
+ base64::engine::general_purpose::STANDARD.encode(&b),
+ )
+ })
+ .unwrap_or(serde_json::Value::Null));
+ }
+ error!(
+ "Column '{column_name}' has unsupported MySQL type '{type_name}', \
+ returning null"
+ );
+ Ok(serde_json::Value::Null)
+ }
+ }
+}
+
+fn value_as_string(value: &serde_json::Value) -> Option<String> {
+ match value {
+ serde_json::Value::String(s) => Some(s.clone()),
+ serde_json::Value::Number(n) => Some(n.to_string()),
+ _ => None,
+ }
+}
+
+fn build_processed_row(
+ payload: Vec<u8>,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+) -> ProcessedRow {
+ let now = Utc::now().timestamp_millis() as u64;
+ ProcessedRow {
+ message: ProducedMessage {
+ id: Some(Uuid::new_v4().as_u128()),
+ headers: None,
+ checksum: None,
+ timestamp: Some(now),
+ origin_timestamp: Some(now),
+ payload,
+ },
+ max_offset,
+ row_pk,
+ }
+}
+
+fn to_snake_case(input: &str) -> String {
+ let mut result = String::new();
+ let mut prev_was_uppercase = false;
+ for (i, ch) in input.chars().enumerate() {
+ if ch.is_uppercase() {
+ if i > 0 && !prev_was_uppercase {
+ result.push('_');
+ }
+ if let Some(lc) = ch.to_lowercase().next() {
+ result.push(lc);
+ } else {
+ result.push(ch);
+ }
+ prev_was_uppercase = true;
+ } else {
+ result.push(ch);
+ prev_was_uppercase = false;
+ }
+ }
+ result
+}
+
+fn redact_connection_string(conn_str: &str) -> String {
+ if let Some(scheme_end) = conn_str.find("://") {
+ let scheme = &conn_str[..scheme_end + 3];
+ let rest = &conn_str[scheme_end + 3..];
+ let preview: String = rest.chars().take(3).collect();
+ return format!("{scheme}{preview}***");
+ }
+ let preview: String = conn_str.chars().take(3).collect();
+ format!("{preview}***")
+}
+
+fn quote_identifier(name: &str) -> Result<String, Error> {
+ if name.is_empty() {
+ return Err(Error::InvalidConfigValue(
+ "identifier must not be empty".to_string(),
+ ));
+ }
+ if name.contains('\0') {
+ return Err(Error::InvalidConfigValue(format!(
+ "identifier '{name}' contains NUL byte"
+ )));
+ }
+ let escaped = name.replace('`', "``");
+ Ok(format!("`{escaped}`"))
+}
+
+fn quote_qualified_identifier(name: &str) -> Result<String, Error> {
+ if !name.contains('.') {
+ return quote_identifier(name);
+ }
+ let parts: Result<Vec<_>, _> = name.split('.').map(quote_identifier).collect();
+ Ok(parts?.join("."))
+}
+
+fn format_offset_value(value: &str) -> String {
+ if value.parse::<i64>().is_ok() || value.parse::<f64>().is_ok_and(|v| v.is_finite()) {
+ value.to_string()
+ } else {
+ let escaped = value
+ .replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "");
+ format!("'{escaped}'")
+ }
+}
+
+fn is_transient_error(e: &sqlx::Error) -> bool {
+ match e {
+ sqlx::Error::Io(_) => true,
+ sqlx::Error::PoolTimedOut => true,
+ sqlx::Error::PoolClosed => false,
+ sqlx::Error::Protocol(_) => false,
+ // MySQL surfaces a numeric error code (e.g. 1213) and a SQLSTATE (e.g. "40001").
+ // `DatabaseError::code()` returns the SQLSTATE, so matching it against MySQL error
+ // numbers never fires. Downcast to the driver error and compare `number()` instead.
+ sqlx::Error::Database(db_err) => db_err
+ .try_downcast_ref::<MySqlDatabaseError>()
+ .is_some_and(|mysql_err| {
+ matches!(
+ mysql_err.number(),
+ // concurrency
+ 1213 | 1205 |
+ // server unavailability
+ 1053 | 1152 | 1080 |
+ // connection/network
+ 2006 | 2013 | 1158 | 1159 | 1160 | 1161 |
Review Comment:
2006/2013 are client-side CR_* codes - sqlx surfaces lost connections as `sqlx::Error::Io`, and `MySqlDatabaseError::number()` only ever carries server ER_* codes, so these two can't match here. harmless, but they suggest coverage that isn't there. 1158-1161 are real server codes and fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on PR #3568:
URL: https://github.com/apache/iggy/pull/3568#issuecomment-4928635969
/author
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3482894567
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1358 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ if self.config.delete_after_read.unwrap_or(false) {
+ let delete_query =
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})");
+
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+
+ sqlx::query(sqlx::AssertSqlSafe(delete_query))
+ .execute(pool)
+ .await
+ .map_err(|e| {
+ error!("Failed to delete processed rows: {e}");
+ Error::InvalidRecord
+ })?;
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ let update_query = format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ );
+
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+
+ sqlx::query(sqlx::AssertSqlSafe(update_query))
+ .execute(pool)
+ .await
+ .map_err(|e| {
+ error!("Failed to mark rows as processed: {e}");
+ Error::InvalidRecord
+ })?;
+ }
+
+ Ok(())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+ let mut messages = Vec::new();
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Collect state updates to apply after processing
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for table in &self.config.tables {
+ let table_config = RowProcessingConfig {
+ table,
+ ..row_config
+ };
+
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.validate_custom_query(custom_query)?;
+ self.substitute_query_params(custom_query, table, &last_offset, batch_size)
+ } else {
+ self.build_polling_query(table, tracking_column, &last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut max_offset: Option<String> = None;
+ let mut processed_ids: Vec<String> = Vec::new();
+
+ let mut count_per_table = 0;
+ for row in rows {
+ let processed = self.process_row(&row, &table_config)?;
+
+ if let Some(pk) = processed.row_pk {
+ processed_ids.push(pk);
+ }
+ if let Some(offset) = processed.max_offset {
+ max_offset = Some(offset);
+ }
+
+ messages.push(processed.message);
+ count_per_table += 1;
+ total_processed += 1;
+ }
+
+ // Database I/O without holding the lock
+ if !processed_ids.is_empty() {
+ self.mark_or_delete_processed_rows(pool, table, pk_column, &processed_ids)
+ .await?;
+ }
Review Comment:
Valid concern — I hit this too. The delete-before-send window exists because the SDK has no post-send ack primitive, not a conscious design choice. I modeled it on postgres_source as the reference, but that's context not a justification. Proper fix needs a post-send hook on the Source trait — SDK-level change, out of scope for this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on PR #3568:
URL: https://github.com/apache/iggy/pull/3568#issuecomment-5427459358
/ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3554102122
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
Review Comment:
**345 vs 377-384** — built queries use format_offset_value() (quotes string offsets); substitute_query_params drops raw $offset. String/timestamp custom query breaks or wrong comparison. Postgres same pattern. **Fix**: pipe $offset through format_offset_value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3603866423
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1563 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ // Skip the pacing delay while draining a backlog
+ if !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ // Idle cycles produce no messages and leave offsets/processed_rows
+ // untouched, so return None to let the runtime skip the state fsync.
+ let persisted_state = if messages.is_empty() {
+ None
+ } else {
+ self.serialize_state(&state)
+ };
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+ let offset = format_offset_value(&offset_value);
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now_unix", &now.timestamp().to_string())
+ .replace("$now", &now.to_rfc3339())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
Review Comment:
748-791,427 — hot path: per-row serde_json Map + SELECT *. **Fix**: column projection + direct serialize (v1 acceptable, matches postgres shape).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on PR #3568:
URL: https://github.com/apache/iggy/pull/3568#issuecomment-5004297540
No major issues. Most of them are suggestions/documentation. Should be good to merge in the next iteration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r4303219889
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,4071 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source,
+ source::SourceBatchResult, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+/// Upper bound on applying a batch's staged mark/delete after its Ack. The SDK waits
+/// `BATCH_RESULT_TIMEOUT` (30s) for the result, so this has to finish well inside it.
+const ACK_BATCH_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// MySQL rejects a prepared statement with more placeholders than this (error 1390), and
+/// the mark/delete binds one per row of the batch.
+const MAX_CLEANUP_BATCH_SIZE: u32 = 65_535;
+
+/// How many mark/delete statements in a row may fail before the connector stops. The
+/// operations stay in the persisted state and run again on restart.
+const MAX_CONSECUTIVE_CLEANUP_FAILURES: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ pending_batch: Mutex<Option<PendingBatch>>,
+ consecutive_cleanup_failures: AtomicU32,
+ verified_cleanup_keys: Mutex<HashSet<String>>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some("it has too few distinct values to be unique per row");
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some("binary values are compared as base64 text, which does not keep byte order");
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some("MySQL orders these by declaration index, not by label");
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL \
+ or not an ordered scalar (tinyint(1), JSON)"
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "row '{value}' is below the stored offset '{previous}' for tracking_column \
+ '{column}'. Filter the query with `{column} > $offset`"
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. Order the query ascending by '{column}'"
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+ /// Mark/delete work for a batch the runtime has already sent. It is persisted with
+ /// the batch's offsets so a crash between the send and the cleanup reruns it.
+ #[serde(default)]
+ pending_operations: Vec<PendingOperation>,
+}
+
+/// The rows of one table to mark or delete once their batch is acknowledged.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct PendingOperation {
+ table: String,
+ ids: Vec<PkValue>,
+ target: CleanupTarget,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct CleanupTarget {
+ key_column: String,
+ action: CleanupAction,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum CleanupAction {
+ Delete,
+ MarkProcessed { column: String },
+}
+
+/// The candidate state of the batch most recently handed to the runtime. Live state
+/// only takes it over on Ack, so a Nack leaves offsets and rows where they were.
+#[derive(Debug, Clone)]
+struct PendingBatch {
+ state: State,
+ /// The runtime has delivered the batch but its cleanup has not finished yet.
+ acknowledged: bool,
+ /// Cleanup finished and the next poll must persist `state` without the operations,
+ /// so a restart does not run them again.
+ retiring: bool,
+}
+
+struct PolledBatch {
+ messages: Vec<ProducedMessage>,
+ pending: Option<PendingBatch>,
+}
+
+enum CleanupKeyCheck {
+ Unique,
+ NotUnique,
+ TableMissing,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty, list the table even when custom_query hardcodes it"
+ .to_string(),
+ ));
+ }
+
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ if let Some(batch_size) = self.config.batch_size
+ && batch_size > MAX_CLEANUP_BATCH_SIZE
+ && self.cleanup_target().is_some()
+ {
+ return Err(Error::InitError(format!(
+ "batch_size {batch_size} exceeds {MAX_CLEANUP_BATCH_SIZE} with delete_after_read or \
+ processed_column set, MySQL allows at most {MAX_CLEANUP_BATCH_SIZE} placeholders \
+ in the statement that marks or deletes a batch"
+ )));
+ }
+
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "name the column custom_query orders by and filters with $offset"
+ } else {
+ "use a unique, monotonically increasing column"
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set, {guidance}"
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.validate_pending_cleanup()?;
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+ self.validate_cleanup_keys().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ if let Some(state) = self.retiring_state().await {
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: Some(self.serialize_pending_state(&state)?),
+ });
+ }
+
+ // Skip the pacing delay while draining a backlog, but never while a cleanup
+ // retry is pending, or a failing mark/delete would spin without a pause.
+ let cleanup_pending = self.pending_batch.lock().await.is_some();
+ if cleanup_pending || !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ {
+ let pending = self.pending_batch.lock().await;
+ if let Some(pending) = pending.as_ref() {
+ if !pending.acknowledged {
+ error!(
+ "{CONNECTOR_NAME} connector ID: {} was polled while a batch was still in flight",
+ self.id
+ );
+ return Err(Error::InvalidState);
+ }
+ // An empty batch whose Ack retries the cleanup the last one left behind.
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: None,
+ });
+ }
+ }
+
+ let polled = self.poll_tables().await?;
+
+ let processed_rows = match &polled.pending {
+ Some(pending) => pending.state.processed_rows,
+ None => self.state.lock().await.processed_rows,
+ };
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ }
+
+ // Idle cycles stage nothing, so return None to let the runtime skip the fsync.
+ let persisted_state = polled
+ .pending
+ .as_ref()
+ .map(|pending| self.serialize_pending_state(&pending.state))
+ .transpose()?;
+ *self.pending_batch.lock().await = polled.pending;
+
+ Ok(ProducedMessages {
+ schema,
+ messages: polled.messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> {
+ if result == SourceBatchResult::Nack {
+ // Dropping the candidate leaves the live offsets where they were, so the
+ // next poll reads the same rows again. Delivered batches keep their cleanup.
+ let mut pending = self.pending_batch.lock().await;
+ if pending
+ .as_ref()
+ .is_some_and(|pending| !pending.acknowledged && !pending.retiring)
+ {
+ pending.take();
+ }
+ return Ok(());
+ }
+
+ let Some(PendingBatch { mut state, .. }) = self.pending_batch.lock().await.clone() else {
+ return Ok(());
+ };
+
+ let operations = std::mem::take(&mut state.pending_operations);
+ let had_operations = !operations.is_empty();
+ let deadline = tokio::time::Instant::now() + ACK_BATCH_TIMEOUT;
+ let failed_operations = self.apply_pending_operations(operations, deadline).await?;
+
+ if failed_operations.is_empty() {
+ *self.state.lock().await = state.clone();
+ let mut pending = self.pending_batch.lock().await;
+ if had_operations {
+ *pending = Some(PendingBatch {
+ state,
+ acknowledged: false,
+ retiring: true,
+ });
+ } else {
+ pending.take();
+ }
+ } else {
+ state.pending_operations = failed_operations;
+ *self.pending_batch.lock().await = Some(PendingBatch {
+ state,
+ acknowledged: true,
+ retiring: false,
+ });
+ }
+
+ Ok(())
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ let state = restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ pending_operations: Vec::new(),
+ });
+ // The runtime only persists a batch's operations after sending it, so restored
+ // operations belong to a delivered batch and only the cleanup is left to do.
+ let pending_batch = (!state.pending_operations.is_empty()).then(|| PendingBatch {
+ state: state.clone(),
+ acknowledged: true,
+ retiring: false,
+ });
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(state),
+ pending_batch: Mutex::new(pending_batch),
+ consecutive_cleanup_failures: AtomicU32::new(0),
+ verified_cleanup_keys: Mutex::new(HashSet::new()),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ poisoned_tables: Mutex::new(HashSet::new()),
+ tracking_kinds: Mutex::new(HashMap::new()),
+ polls: AtomicU64::new(0),
+ cursor_regressions: Mutex::new(HashMap::new()),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ // `format_offset_value` escapes backslashes, which under NO_BACKSLASH_ESCAPES
+ // would reach MySQL as literal characters and filter on the wrong cursor.
+ .after_connect(|connection, _| {
+ Box::pin(async move {
+ sqlx::query(
+ "SET SESSION sql_mode = REPLACE(@@SESSION.sql_mode, 'NO_BACKSLASH_ESCAPES', '')",
+ )
+ .execute(&mut *connection)
+ .await?;
+ Ok(())
+ })
+ })
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn tracking_column(&self) -> &str {
+ self.config.tracking_column.as_deref().unwrap_or("id")
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn resolve_tracking_columns(&self) -> Result<HashMap<String, OffsetKind>, Error> {
+ // Whether the connector owns the query, and can therefore hold the schema to
+ // what that query will do with it.
+ let strict = self.config.custom_query.is_none();
+ let pool = self.get_pool()?;
+ let tracking_column = self.tracking_column();
+ let mut kinds = HashMap::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ }
+
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ }
+
+ warn!(
+ "tracking_column '{tracking_column}' is not a column of table '{table}'; \
+ assuming custom_query projects it. Its comparison order is resolved from \
+ the first result set instead."
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ // The column's type survives a plain projection, so it is worth rejecting
+ // whatever the query looks like. A custom_query aliasing an expression onto
+ // a name the table also uses is the one false positive, hence the hint.
+ if let Some(reason) = unusable_tracking_type(&data_type) {
+ let aliasing_hint = if self.config.custom_query.is_some() {
+ " If custom_query aliases a different expression to this name, rename the \
+ alias so it does not collide with the column."
+ } else {
+ ""
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': {reason}. Use an integer, timestamp, or string \
+ column.{aliasing_hint}"
+ )));
+ }
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; a row \
+ whose value is NULL cannot produce a cursor and will fail that table's batch. \
+ Have custom_query exclude those rows, or declare the column NOT NULL."
+ );
+ }
+
+ if is_collation_ordered(&data_type) {
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': values must be unique and increase in its collation order \
+ (zero-pad numbers, prefer a _bin collation)"
+ );
+ }
+
+ kinds.insert(table.clone(), offset_kind_for_data_type(&data_type));
+ }
+ Ok(kinds)
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn serialize_pending_state(&self, state: &State) -> Result<ConnectorState, Error> {
+ self.serialize_state(state).ok_or_else(|| {
+ Error::Serialization(format!("failed to serialize {CONNECTOR_NAME} state"))
+ })
+ }
+
+ async fn retiring_state(&self) -> Option<State> {
+ self.pending_batch
+ .lock()
+ .await
+ .as_ref()
+ .filter(|pending| pending.retiring)
+ .map(|pending| pending.state.clone())
+ }
+
+ fn primary_key_column(&self) -> &str {
+ self.config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(self.tracking_column())
+ }
+
+ fn cleanup_target(&self) -> Option<CleanupTarget> {
+ let action = if self.config.delete_after_read.unwrap_or(false) {
+ CleanupAction::Delete
+ } else if let Some(column) = &self.config.processed_column {
+ CleanupAction::MarkProcessed {
+ column: column.clone(),
+ }
+ } else {
+ return None;
+ };
+ Some(CleanupTarget {
+ key_column: self.primary_key_column().to_string(),
+ action,
+ })
+ }
+
+ /// Restored operations were staged under the config of the run that wrote them.
+ /// Running them under a different key or action would touch rows nobody read.
+ fn validate_pending_cleanup(&mut self) -> Result<(), Error> {
+ let configured = self.cleanup_target();
+ for operation in &self.state.get_mut().pending_operations {
+ if configured.as_ref() != Some(&operation.target) {
+ return Err(Error::InitError(format!(
+ "the cleanup configuration changed while cleanup of {} delivered row(s) in \
+ table '{}' is pending; restore primary_key_column '{}' and action {:?}",
+ operation.ids.len(),
+ operation.table,
+ operation.target.key_column,
+ operation.target.action
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ /// `WHERE key IN (...)` on a key that repeats marks or deletes every row sharing a
+ /// value, including ones that were never read. Tables that don't exist yet are
+ /// checked by the first poll that stages cleanup for them.
+ async fn validate_cleanup_keys(&mut self) -> Result<(), Error> {
+ let Some(target) = self.cleanup_target() else {
+ return Ok(());
+ };
+ let pool = self.get_pool()?;
+ let mut verified = HashSet::with_capacity(self.config.tables.len());
+ for table in &self.config.tables {
+ let check = cleanup_key_check(pool, table, &target.key_column)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to check primary_key_column '{}' on table '{table}': {e}",
+ target.key_column
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ verified.insert(table.clone());
+ }
+ CleanupKeyCheck::NotUnique => {
+ return Err(Error::InitError(not_unique_key_message(
+ table,
+ &target.key_column,
+ )));
+ }
+ CleanupKeyCheck::TableMissing => {}
+ }
+ }
+ *self.verified_cleanup_keys.get_mut() = verified;
+ Ok(())
+ }
+
+ async fn cleanup_key_is_unique(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ key_column: &str,
+ ) -> Result<bool, Error> {
+ if self.verified_cleanup_keys.lock().await.contains(table) {
+ return Ok(true);
+ }
+ let check = cleanup_key_check(pool, table, key_column)
+ .await
+ .map_err(|e| {
+ Error::Connection(format!(
+ "failed to check primary_key_column '{key_column}' on table '{table}': {e}"
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ self.verified_cleanup_keys
+ .lock()
+ .await
+ .insert(table.to_string());
+ Ok(true)
+ }
+ CleanupKeyCheck::NotUnique => Ok(false),
+ CleanupKeyCheck::TableMissing => Err(Error::Connection(format!(
+ "table '{table}' disappeared before its primary_key_column could be checked"
+ ))),
+ }
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ /// Best current knowledge of how a table's tracking values compare.
+ async fn tracking_kind(&self, table: &str) -> OffsetKind {
+ self.tracking_kinds
+ .lock()
+ .await
+ .get(table)
+ .copied()
+ .unwrap_or(OffsetKind::Unknown)
+ }
+
+ /// Whether the stored cursor is what limits the rows a poll can see. The built
+ /// polling query always filters on it; a `custom_query` does so only if it
+ /// interpolates `$offset`.
+ fn cursor_gates_query(&self) -> bool {
+ match &self.config.custom_query {
+ Some(query) => query.contains("$offset"),
+ None => true,
+ }
+ }
+
+ /// The offset a poll actually resumes from: the stored cursor, or the configured
+ /// starting point before anything has been stored.
+ fn effective_offset(&self, last_offset: &Option<String>) -> Option<String> {
+ last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let offset_value = self.effective_offset(last_offset).unwrap_or_default();
+ let offset = format_offset_value(&offset_value, kind);
+ let quoted_table = quote_qualified_identifier(table)?;
+ let limit = batch_size.to_string();
+ let now = Utc::now();
+ let now_unix = now.timestamp().to_string();
+ let now = now.to_rfc3339();
+ let replacements = [
+ ("$table", quoted_table.as_str()),
+ ("$offset", offset.as_str()),
+ ("$limit", limit.as_str()),
+ ("$now_unix", now_unix.as_str()),
+ ("$now", now.as_str()),
+ ];
+
+ Ok(substitute_query_tokens(query, &replacements))
+ }
+
+ /// The query is substituted as text and re-executed once per configured table on
+ /// every poll for the life of the process, so a shape that cannot terminate does
+ /// not fail, it repeats. What can be judged from the text alone is judged here,
+ /// at startup, rather than left to the first poll.
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let tracking_column = self.tracking_column();
+
+ match leading_statement_token(query) {
+ None => {
+ return Err(Error::InitError(
+ "custom_query has no statement: it is empty, or nothing but comments"
+ .to_string(),
+ ));
+ }
+ Some(token) if !READ_STATEMENT_STARTS.contains(&token.as_str()) => {
+ return Err(Error::InitError(format!(
+ "custom_query must be a read starting with SELECT, WITH or TABLE, found \
+ '{token}'"
+ )));
+ }
+ Some(_) => {}
+ }
+
+ // Collapsed so `ORDER\nBY` and `ORDER BY` read the same as `ORDER BY`; the
+ // check hard-fails, so a formatting difference must not decide it.
+ let normalized = query
+ .split_whitespace()
+ .collect::<Vec<_>>()
+ .join(" ")
+ .to_uppercase();
+
+ if query.contains("$offset") {
+ if !normalized.contains("ORDER BY") {
+ return Err(Error::InitError(format!(
+ "custom_query filters on $offset but has no ORDER BY, add \
+ `ORDER BY {tracking_column}`"
+ )));
+ }
+ } else if !self.config.delete_after_read.unwrap_or(false)
+ && self.config.processed_column.is_none()
+ {
+ return Err(Error::InitError(format!(
+ "custom_query must filter with $offset (`WHERE {tracking_column} > $offset`), or \
+ set delete_after_read or processed_column"
+ )));
+ }
+
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "custom_query uses $table but no tables are configured".to_string(),
+ ));
+ }
+ // Without $table the same SQL is executed once per table, so identical rows
+ // would be emitted repeatedly, each carrying a different table's metadata,
+ // message ID, and tracking offset. Table identity is not recoverable from a
+ // static query, so reject the combination instead of guessing one.
+ if !query.contains("$table") && self.config.tables.len() > 1 {
+ return Err(Error::InitError(format!(
+ "custom_query has no $table placeholder but {} tables are configured ({}), use \
+ $table in the query or configure a single table",
+ self.config.tables.len(),
+ self.config.tables.join(", ")
+ )));
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = self.effective_offset(last_offset) {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(&offset, kind)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn apply_pending_operations(
+ &self,
+ operations: Vec<PendingOperation>,
+ deadline: tokio::time::Instant,
+ ) -> Result<Vec<PendingOperation>, Error> {
+ let mut failed_operations = Vec::new();
+ let mut operations = operations.into_iter();
+ while let Some(operation) = operations.next() {
+ if tokio::time::Instant::now() >= deadline {
+ failed_operations.push(operation);
+ failed_operations.extend(operations);
+ break;
+ }
+ let result = tokio::time::timeout_at(deadline, async {
+ let pool = self.get_pool()?;
+ self.mark_or_delete_processed_rows(pool, &operation).await
+ })
+ .await;
+ match result {
+ Ok(Ok(())) => {
+ self.consecutive_cleanup_failures
+ .store(0, Ordering::Relaxed);
+ }
+ Ok(Err(error)) => {
+ self.record_cleanup_failure(&operation, &error)?;
+ failed_operations.push(operation);
+ }
+ Err(_) => {
+ self.record_cleanup_failure(
+ &operation,
+ "cleanup exceeded the 10s budget for one Ack",
+ )?;
+ failed_operations.push(operation);
+ }
+ }
+ }
+ Ok(failed_operations)
+ }
+
+ fn record_cleanup_failure(
+ &self,
+ operation: &PendingOperation,
+ error: impl std::fmt::Display,
+ ) -> Result<(), Error> {
+ let consecutive_failures = self
+ .consecutive_cleanup_failures
+ .fetch_add(1, Ordering::Relaxed)
+ .saturating_add(1);
+ error!(
+ "Failed to mark or delete {} delivered row(s) in table '{}' for {CONNECTOR_NAME} \
+ connector ID: {}, retrying on the next poll ({consecutive_failures} consecutive \
+ failures): {error}",
+ operation.ids.len(),
+ operation.table,
+ self.id
+ );
+ if consecutive_failures >= MAX_CONSECUTIVE_CLEANUP_FAILURES {
+ return Err(Error::Connection(format!(
+ "stopping {CONNECTOR_NAME} connector ID {} after {consecutive_failures} \
+ consecutive mark/delete failures; the cleanup stays in the saved state and \
+ runs again on restart",
+ self.id
+ )));
+ }
+ Ok(())
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ operation: &PendingOperation,
+ ) -> Result<(), Error> {
+ let PendingOperation { table, ids, target } = operation;
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(&target.key_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = match &target.action {
+ CleanupAction::Delete => {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
+ }
+ CleanupAction::MarkProcessed { column } => {
+ let quoted_processed = quote_identifier(column)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({placeholders})"
+ )
+ }
+ };
+
+ // The query and its binds are rebuilt on every attempt because with_retry
+ // calls the closure once per try and each `execute` consumes the query.
+ with_retry(
+ || {
+ let mut prepared = sqlx::query(sqlx::AssertSqlSafe(query.as_str()));
+ for id in ids {
+ prepared = match id {
+ PkValue::Text(text) => prepared.bind(text),
+ PkValue::Bytes(bytes) => prepared.bind(bytes),
+ };
+ }
+ prepared.execute(pool)
+ },
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<PolledBatch, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.tracking_column();
+ let pk_column = self.primary_key_column();
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ //
+ // An ordering violation is handled one step further. Retrying it would
+ // re-run the same query, re-decode the same rows, and hit the same failure
+ // forever, so the table is latched off instead of being retried each cycle.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ let poll_number = self.polls.fetch_add(1, Ordering::Relaxed) + 1;
+ let poisoned = { self.poisoned_tables.lock().await.clone() };
+ let mut newly_poisoned: Vec<String> = Vec::new();
+
+ for table in &self.config.tables {
+ if poisoned.contains(table) {
+ continue;
+ }
+
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ if !batch.messages.is_empty() {
+ self.cursor_regressions.lock().await.remove(table);
+ }
+ // A full batch means the LIMIT was hit, so more rows are waiting.
+ any_batch_full |= batch.messages.len() as u32 >= batch_size;
+ batches.push(batch);
+ }
+ Err(FetchError::Ordering(reason) | FetchError::MissingTrackingColumn(reason)) => {
+ error!(
+ "Table '{table}' at offset {}: {reason}. Table disabled until restart.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ newly_poisoned.push(table.clone());
+ }
+ Err(FetchError::CursorRegression(reason)) => {
+ // Cleared only by a batch that carried rows, so the run counts polls
+ // since the table last published anything. An idle poll produces as
+ // little as a rejected one does and must not read as recovery.
+ let consecutive = {
+ let mut regressions = self.cursor_regressions.lock().await;
+ let consecutive = regressions.entry(table.clone()).or_insert(0);
+ *consecutive += 1;
+ *consecutive
+ };
+ let offset = last_offset.as_deref().unwrap_or("<start>");
+ if consecutive >= CURSOR_REGRESSION_ERROR_THRESHOLD {
+ error!(
+ "Table '{table}' at offset {offset}: {reason}. Skipping this poll \
+ ({consecutive} consecutive polls)."
+ );
+ } else {
+ warn!("Table '{table}' at offset {offset}: {reason}. Skipping this poll.");
+ }
+ }
+ Err(FetchError::StaleOffsetKind(reason)) => {
+ warn!(
+ "Table '{table}' at offset {}: {reason}. Retrying on the next poll.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::UnusableTrackingValue(reason)) => {
+ error!(
+ "Table '{table}' at offset {}: {reason}. Retrying each poll until the row \
+ is fixed.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::Other(error)) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ if !newly_poisoned.is_empty() {
+ let mut poisoned_tables = self.poisoned_tables.lock().await;
+ poisoned_tables.extend(newly_poisoned);
+ if self
+ .config
+ .tables
+ .iter()
+ .all(|table| poisoned_tables.contains(table))
+ {
+ error!(
+ "All configured tables are disabled because their batches cannot yield a \
+ cursor; {CONNECTOR_NAME} connector with ID: {} is idle until restarted",
+ self.id
+ );
+ }
+ }
+
+ // From the next poll on a disabled table is skipped before anything is logged,
+ // so the one line written when it was disabled can scroll out of the operator's
+ // log window and leave a dark connector looking healthy.
+ if poll_number.is_multiple_of(DISABLED_TABLES_LOG_INTERVAL) {
+ let poisoned_tables = self.poisoned_tables.lock().await;
+ if !poisoned_tables.is_empty() {
+ let mut disabled: Vec<&str> = poisoned_tables.iter().map(String::as_str).collect();
+ disabled.sort_unstable();
+ error!(
+ "{} of {} configured table(s) are still disabled and are not being polled by \
+ {CONNECTOR_NAME} connector with ID: {}, and will stay that way until it is \
+ restarted: {}",
+ disabled.len(),
+ self.config.tables.len(),
+ self.id,
+ disabled.join(", ")
+ );
+ }
+ }
+
+ // Phase 2: stage each table's batch. Nothing touches MySQL or the live state
+ // here: the offsets and the mark/delete are applied in `on_batch_result` once
+ // the runtime has sent the batch and saved this candidate state, and a Nack
+ // throws the candidate away so the same rows are read again.
+ let cleanup_target = self.cleanup_target();
+ let mut candidate_state = self.state.lock().await.clone();
+ let mut operations = Vec::new();
+ let mut messages = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if let Some(target) = &cleanup_target
+ && !batch.messages.is_empty()
+ {
+ if batch.processed_ids.is_empty() {
+ error!(
+ "Table '{}': mark/delete is configured but no primary keys were extracted \
+ from {} row(s), skipping this cycle so rows are not published without \
+ being marked or deleted (check that '{pk_column}' is projected and scalar)",
+ batch.table,
+ batch.messages.len()
+ );
+ continue;
+ }
+
+ match self
+ .cleanup_key_is_unique(pool, &batch.table, &target.key_column)
+ .await
+ {
+ Ok(true) => {}
+ Ok(false) => {
+ error!(
+ "{}. Table disabled until restart.",
+ not_unique_key_message(&batch.table, &target.key_column)
+ );
+ self.poisoned_tables.lock().await.insert(batch.table);
+ continue;
+ }
+ Err(error) => {
+ error!("Table '{}' skipped this cycle: {error}", batch.table);
+ continue;
+ }
+ }
+
+ operations.push(PendingOperation {
+ table: batch.table.clone(),
+ ids: std::mem::take(&mut batch.processed_ids),
+ target: target.clone(),
+ });
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ candidate_state.tracking_offsets.insert(batch.table, offset);
+ }
+ }
+
+ let pending = (total_processed > 0).then(|| {
+ candidate_state.processed_rows += total_processed;
+ candidate_state.pending_operations = operations;
+ PendingBatch {
+ state: candidate_state,
+ acknowledged: false,
+ retiring: false,
+ }
+ });
+
+ self.last_batch_full
+ .store(any_batch_full, Ordering::Relaxed);
+ Ok(PolledBatch { messages, pending })
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, FetchError> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query_kind = self.tracking_kind(table).await;
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.substitute_query_params(custom_query, table, last_offset, batch_size, query_kind)?
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size, query_kind)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ // Settled once for the whole batch rather than per row, since the column list
+ // is a property of the result set.
+ if let Some(first) = rows.first() {
+ ensure_tracking_column_projected(
+ first.columns().iter().map(|column| column.name()),
+ tracking_column,
+ )?;
+ }
+
+ let resume_from = if self.cursor_gates_query() {
+ self.effective_offset(last_offset)
+ } else {
+ None
+ };
+
+ // The result set describes what this query actually returned, so it outranks
+ // whatever `information_schema` said at startup and is the only answer
+ // available for a computed column.
+ let kind = match rows.first().and_then(|row| {
+ observed_tracking_kind(row, tracking_column).filter(|observed| *observed != query_kind)
+ }) {
+ Some(observed) => {
+ self.tracking_kinds
+ .lock()
+ .await
+ .insert(table.to_string(), observed);
+ // The query was already sent with the stale kind, so its offset literal
+ // may be quoted for the wrong type.
+ if let Some(offset) = &resume_from
+ && offset_literal_diverged(offset, query_kind, observed)
+ {
+ return Err(FetchError::StaleOffsetKind(format!(
+ "tracking_column '{tracking_column}' type differs from the one detected \
+ at startup (filtered with {}, expected {}), batch discarded",
+ format_offset_value(offset, query_kind),
+ format_offset_value(offset, observed)
+ )));
+ }
+ observed
+ }
+ None => query_kind,
+ };
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ // Every row of a result set shares its columns, so the published key names are
+ // derived once per batch instead of once per row.
+ let column_names: Vec<String> = rows
+ .first()
+ .map(|row| {
+ row.columns()
+ .iter()
+ .map(|column| {
+ if table_config.snake_case_columns {
+ to_snake_case(column.name())
+ } else {
+ column.name().to_string()
+ }
+ })
+ .collect()
+ })
+ .unwrap_or_default();
+ let mut ordering = OrderingGuard::new(tracking_column, kind, resume_from);
+ for row in rows {
+ let processed = self
+ .process_row(&row, &table_config, &column_names)
+ .map_err(|e| {
+ error!(
+ "Failed to decode row in table '{table}' (pk {}): {e}",
+ pk_for_log(&row, table_config.pk_column)
+ );
+ e
+ })?;
+
+ let Some(value) = processed.tracking_value else {
+ return Err(unusable_tracking_value(
+ tracking_column,
+ &pk_for_log(&row, table_config.pk_column),
+ ));
+ };
+ ordering
+ .accept(value)
+ .map_err(|violation| match violation {
+ OrderingViolation::WithinBatch(reason) => FetchError::Ordering(reason),
+ OrderingViolation::BelowCursor(reason) => FetchError::CursorRegression(reason),
+ })?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+
+ batch.messages.push(processed.message);
+ }
+ batch.max_offset = ordering.into_cursor();
+
+ Ok(batch)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ column_names: &[String],
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<PkValue> = None;
+ let mut tracking_value: Option<String> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
+ let mut extracted_payload: Option<Vec<u8>> = None;
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ if name.eq_ignore_ascii_case(config.payload_col) {
+ extracted_payload =
+ Some(self.extract_payload_column(row, i, config.payload_format)?);
+ }
+ let is_tracking = name.eq_ignore_ascii_case(config.tracking_column);
+ let is_pk = name.eq_ignore_ascii_case(config.pk_column);
+ if is_tracking {
+ let value = extract_column_value(row, i)?;
+ tracking_value = column_value_as_string(row, i, &value)?;
+ if is_pk {
+ row_pk = pk_value_from_decoded(row, i, &value)?;
+ }
+ } else if is_pk {
+ row_pk = extract_pk_value(row, i)?;
+ }
+ }
+
+ // Every extract_payload_column arm yields Ok for a NULL column, so None
+ // here means the column was absent. Falling back to the whole-row JSON
+ // would publish bytes contradicting the schema derived from payload_format.
+ let payload = extracted_payload.ok_or_else(|| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': payload_column '{}' not present in the result set",
+ config.table, config.payload_col
+ ))
+ })?;
+
+ return Ok(build_processed_row(
+ config.table,
+ payload,
+ tracking_value,
+ row_pk,
+ ));
+ }
+
+ let mut data = serde_json::Map::new();
+ for ((i, column), column_name) in row.columns().iter().enumerate().zip(column_names) {
+ let name = column.name();
+ let value = extract_column_value(row, i)?;
+ if name.eq_ignore_ascii_case(config.tracking_column) {
+ tracking_value = column_value_as_string(row, i, &value)?;
+ }
+ if name.eq_ignore_ascii_case(config.pk_column) {
+ row_pk = pk_value_from_decoded(row, i, &value)?;
+ }
+ data.insert(column_name.clone(), value);
+ }
+
+ let payload = if config.include_metadata {
+ let record = DatabaseRecord {
+ table_name: config.table.to_string(),
+ operation_type: "SELECT".to_string(),
+ timestamp: Utc::now(),
+ data: serde_json::Value::Object(data),
+ old_data: None,
+ };
+ simd_json::to_vec(&record).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': failed to serialize row to JSON: {e}",
+ config.table
+ ))
+ })?
+ } else {
+ simd_json::to_vec(&data).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': failed to serialize row to JSON: {e}",
+ config.table
+ ))
+ })?
+ };
+
+ Ok(build_processed_row(
+ config.table,
+ payload,
+ tracking_value,
+ row_pk,
+ ))
+ }
+}
+
+/// Type-faithful primary-key extraction. Binary columns are taken as raw bytes
+/// rather than routed through the base64 path in `extract_column_value`, so the
+/// value can be bound back into `WHERE pk IN (...)` and match the row. Everything
+/// else takes the string form MySQL parses back to the same value. Returns None
+/// only for a NULL pk, which a real primary key cannot be.
+fn extract_pk_value(row: &MySqlRow, column_index: usize) -> Result<Option<PkValue>, Error> {
+ if is_binary_column(row, column_index) {
+ return binary_pk_value(row, column_index);
+ }
+ let value = extract_column_value(row, column_index)?;
+ pk_value_from_decoded(row, column_index, &value)
+}
+
+/// Like `extract_pk_value` for a column the caller has already decoded, so the
+/// common key column is not decoded a second time per row.
+fn pk_value_from_decoded(
+ row: &MySqlRow,
+ column_index: usize,
+ value: &serde_json::Value,
+) -> Result<Option<PkValue>, Error> {
+ if is_binary_column(row, column_index) {
+ return binary_pk_value(row, column_index);
+ }
+ Ok(column_value_as_string(row, column_index, value)?.map(PkValue::Text))
+}
+
+fn is_binary_column(row: &MySqlRow, column_index: usize) -> bool {
+ matches!(
+ row.columns()[column_index].type_info().name(),
+ "BINARY" | "VARBINARY" | "TINYBLOB" | "BLOB" | "MEDIUMBLOB" | "LONGBLOB"
+ )
+}
+
+fn binary_pk_value(row: &MySqlRow, column_index: usize) -> Result<Option<PkValue>, Error> {
+ let column = &row.columns()[column_index];
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "primary key column '{}' (MySQL type '{}'): {e}",
+ column.name(),
+ column.type_info().name()
+ ))
+ })?;
+ Ok(bytes.map(PkValue::Bytes))
+}
+
+fn extract_column_value(row: &MySqlRow, column_index: usize) -> Result<serde_json::Value, Error> {
+ let column = &row.columns()[column_index];
+ let type_name = column.type_info().name();
+
+ let to_err = |e: sqlx::Error| {
+ Error::InvalidRecordValue(format!(
+ "column '{}' (MySQL type '{type_name}'): {e}",
+ column.name()
+ ))
+ };
+
+ match type_name {
+ "BOOLEAN" => {
+ let value: Option<bool> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::Bool)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT" => {
+ let value: Option<i8> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT" => {
+ let value: Option<i16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT" | "INT" => {
+ let value: Option<i32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT" => {
+ let value: Option<i64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT UNSIGNED" => {
+ let value: Option<u8> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT UNSIGNED" => {
+ let value: Option<u16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT UNSIGNED" | "INT UNSIGNED" => {
+ let value: Option<u32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT UNSIGNED" => {
+ let value: Option<u64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "FLOAT" => {
+ let value: Option<f32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as f64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DOUBLE" => {
+ let value: Option<f64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DECIMAL" => {
+ let value: Option<rust_decimal::Decimal> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|d| serde_json::Value::String(d.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIT" => {
+ let value: Option<u64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "YEAR" => {
+ let value: Option<u16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATE" => {
+ let value: Option<NaiveDate> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|d| serde_json::Value::String(d.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TIME" => {
+ let value: Option<NaiveTime> = row.try_get(column_index).map_err(to_err)?;
Review Comment:
Confirmed and fixed.
Reproduced against MySQL 8: decoding `TIME` as `NaiveTime` dropped the sign (-01:30:00 became 01:30:00) and failed on anything from 24h up, which failed the batch on every poll and stalled the table. It now decodes as `MySqlTime`, so negative values and the full ±838h range are published.
Added `time_column_publishes_negative_and_multi_day_values`, which fails on the previous commit and passes now.
While tracing this I also found that TIME was accepted as a `tracking_column` but compared as text, so correctly ordered values like 9:59:59 → 10:00:00 would trip the ordering guard. `TIME` can't be unique and increasing anyway (it's a repeating time of day or a duration), so it is now rejected as a tracking column like ENUM/JSON, and the README is updated to match.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3603847065
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1563 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ // Skip the pacing delay while draining a backlog
+ if !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ // Idle cycles produce no messages and leave offsets/processed_rows
+ // untouched, so return None to let the runtime skip the state fsync.
+ let persisted_state = if messages.is_empty() {
+ None
+ } else {
+ self.serialize_state(&state)
+ };
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+ let offset = format_offset_value(&offset_value);
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now_unix", &now.timestamp().to_string())
+ .replace("$now", &now.to_rfc3339())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() || id.parse::<u64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).execute(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ // A full batch means the LIMIT was hit, so more rows are waiting.
+ any_batch_full |= batch.messages.len() as u32 >= batch_size;
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. If the mark/delete call fails
+ // for a table we just skip its messages this cycle instead of queuing them
+ // - `mark_or_delete_processed_rows` already retries transient errors, so a
+ // failure here is permanent and the table's rows stay in MySQL to be
+ // retried next cycle, without blocking the others.
+ //
+ // Doesn't cover the case where mark/delete succeeds but the runtime never
+ // gets to publish the batch (crash, send failure, etc) - those rows are
+ // gone from MySQL with nothing delivered. That's just how
+ // delete_after_read/processed_column work, not something this loop fixes.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ let mark_or_delete = self.config.delete_after_read.unwrap_or(false)
+ || self.config.processed_column.is_some();
+
+ for mut batch in batches {
+ if mark_or_delete && !batch.messages.is_empty() && batch.processed_ids.is_empty() {
Review Comment:
mark/delete configured but empty processed_ids: warn then still emit + advance offset. **Fix**: fail batch when mark configured without scalar PK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3479575148
##########
.gitignore:
##########
@@ -42,3 +42,5 @@ go.work
core/bench/dashboard/frontend/dist
LICENSE-binary
**/LICENSE-binary
+local_test_connectors
+local_test_runtime.toml
Review Comment:
i dont see this file being added anywhere in the test in this PR nor in iggy as a whole. please drop it and add it your own `.gitignore_global`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3482894567
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1358 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ if self.config.delete_after_read.unwrap_or(false) {
+ let delete_query =
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})");
+
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+
+ sqlx::query(sqlx::AssertSqlSafe(delete_query))
+ .execute(pool)
+ .await
+ .map_err(|e| {
+ error!("Failed to delete processed rows: {e}");
+ Error::InvalidRecord
+ })?;
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ let update_query = format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ );
+
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+
+ sqlx::query(sqlx::AssertSqlSafe(update_query))
+ .execute(pool)
+ .await
+ .map_err(|e| {
+ error!("Failed to mark rows as processed: {e}");
+ Error::InvalidRecord
+ })?;
+ }
+
+ Ok(())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+ let mut messages = Vec::new();
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Collect state updates to apply after processing
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for table in &self.config.tables {
+ let table_config = RowProcessingConfig {
+ table,
+ ..row_config
+ };
+
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.validate_custom_query(custom_query)?;
+ self.substitute_query_params(custom_query, table, &last_offset, batch_size)
+ } else {
+ self.build_polling_query(table, tracking_column, &last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut max_offset: Option<String> = None;
+ let mut processed_ids: Vec<String> = Vec::new();
+
+ let mut count_per_table = 0;
+ for row in rows {
+ let processed = self.process_row(&row, &table_config)?;
+
+ if let Some(pk) = processed.row_pk {
+ processed_ids.push(pk);
+ }
+ if let Some(offset) = processed.max_offset {
+ max_offset = Some(offset);
+ }
+
+ messages.push(processed.message);
+ count_per_table += 1;
+ total_processed += 1;
+ }
+
+ // Database I/O without holding the lock
+ if !processed_ids.is_empty() {
+ self.mark_or_delete_processed_rows(pool, table, pk_column, &processed_ids)
+ .await?;
+ }
Review Comment:
valid concern, I hit this too. The delete-before-send window exists because the SDK has no post-send ack primitive, not a conscious design choice. I modeled it on postgres_source as the reference, but that's context not a justification.
Proper fix needs a post-send hook on the Source trait — a SDK-level change. Happy to open a separate issue for it if useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3568:
URL: https://github.com/apache/iggy/pull/3568#issuecomment-5247581018
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.
If you need a review, please ensure CI is green and the PR is rebased on the latest master. Don't hesitate to ping the maintainers - either `@core` on Discord or by mentioning them directly here on the PR.
Thank you for your contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3554082355
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).execute(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. A table's messages are emitted
+ // only after its rows are marked or deleted, so a failure here can never
+ // leave a table marked-but-unpublished. `mark_or_delete_processed_rows`
+ // retries transient errors; a permanent failure isolates that one table
+ // (its rows stay in MySQL and are retried next cycle) while the others
+ // still flow.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
Review Comment:
Phase2 mark_or_delete runs before messages leave connector. delete_after_read / processed_column + send fail = row gone/filtered in MySQL, not in Iggy. Same as postgres_source. **Fix**: sibling issue + README hazard note.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r4297516046
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,4071 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source,
+ source::SourceBatchResult, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+/// Upper bound on applying a batch's staged mark/delete after its Ack. The SDK waits
+/// `BATCH_RESULT_TIMEOUT` (30s) for the result, so this has to finish well inside it.
+const ACK_BATCH_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// MySQL rejects a prepared statement with more placeholders than this (error 1390), and
+/// the mark/delete binds one per row of the batch.
+const MAX_CLEANUP_BATCH_SIZE: u32 = 65_535;
+
+/// How many mark/delete statements in a row may fail before the connector stops. The
+/// operations stay in the persisted state and run again on restart.
+const MAX_CONSECUTIVE_CLEANUP_FAILURES: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ pending_batch: Mutex<Option<PendingBatch>>,
+ consecutive_cleanup_failures: AtomicU32,
+ verified_cleanup_keys: Mutex<HashSet<String>>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some("it has too few distinct values to be unique per row");
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some("binary values are compared as base64 text, which does not keep byte order");
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some("MySQL orders these by declaration index, not by label");
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL \
+ or not an ordered scalar (tinyint(1), JSON)"
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "row '{value}' is below the stored offset '{previous}' for tracking_column \
+ '{column}'. Filter the query with `{column} > $offset`"
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. Order the query ascending by '{column}'"
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+ /// Mark/delete work for a batch the runtime has already sent. It is persisted with
+ /// the batch's offsets so a crash between the send and the cleanup reruns it.
+ #[serde(default)]
+ pending_operations: Vec<PendingOperation>,
+}
+
+/// The rows of one table to mark or delete once their batch is acknowledged.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct PendingOperation {
+ table: String,
+ ids: Vec<PkValue>,
+ target: CleanupTarget,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct CleanupTarget {
+ key_column: String,
+ action: CleanupAction,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum CleanupAction {
+ Delete,
+ MarkProcessed { column: String },
+}
+
+/// The candidate state of the batch most recently handed to the runtime. Live state
+/// only takes it over on Ack, so a Nack leaves offsets and rows where they were.
+#[derive(Debug, Clone)]
+struct PendingBatch {
+ state: State,
+ /// The runtime has delivered the batch but its cleanup has not finished yet.
+ acknowledged: bool,
+ /// Cleanup finished and the next poll must persist `state` without the operations,
+ /// so a restart does not run them again.
+ retiring: bool,
+}
+
+struct PolledBatch {
+ messages: Vec<ProducedMessage>,
+ pending: Option<PendingBatch>,
+}
+
+enum CleanupKeyCheck {
+ Unique,
+ NotUnique,
+ TableMissing,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty, list the table even when custom_query hardcodes it"
+ .to_string(),
+ ));
+ }
+
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ if let Some(batch_size) = self.config.batch_size
+ && batch_size > MAX_CLEANUP_BATCH_SIZE
+ && self.cleanup_target().is_some()
+ {
+ return Err(Error::InitError(format!(
+ "batch_size {batch_size} exceeds {MAX_CLEANUP_BATCH_SIZE} with delete_after_read or \
+ processed_column set, MySQL allows at most {MAX_CLEANUP_BATCH_SIZE} placeholders \
+ in the statement that marks or deletes a batch"
+ )));
+ }
+
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "name the column custom_query orders by and filters with $offset"
+ } else {
+ "use a unique, monotonically increasing column"
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set, {guidance}"
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.validate_pending_cleanup()?;
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+ self.validate_cleanup_keys().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ if let Some(state) = self.retiring_state().await {
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: Some(self.serialize_pending_state(&state)?),
+ });
+ }
+
+ // Skip the pacing delay while draining a backlog, but never while a cleanup
+ // retry is pending, or a failing mark/delete would spin without a pause.
+ let cleanup_pending = self.pending_batch.lock().await.is_some();
+ if cleanup_pending || !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ {
+ let pending = self.pending_batch.lock().await;
+ if let Some(pending) = pending.as_ref() {
+ if !pending.acknowledged {
+ error!(
+ "{CONNECTOR_NAME} connector ID: {} was polled while a batch was still in flight",
+ self.id
+ );
+ return Err(Error::InvalidState);
+ }
+ // An empty batch whose Ack retries the cleanup the last one left behind.
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: None,
+ });
+ }
+ }
+
+ let polled = self.poll_tables().await?;
+
+ let processed_rows = match &polled.pending {
+ Some(pending) => pending.state.processed_rows,
+ None => self.state.lock().await.processed_rows,
+ };
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ }
+
+ // Idle cycles stage nothing, so return None to let the runtime skip the fsync.
+ let persisted_state = polled
+ .pending
+ .as_ref()
+ .map(|pending| self.serialize_pending_state(&pending.state))
+ .transpose()?;
+ *self.pending_batch.lock().await = polled.pending;
+
+ Ok(ProducedMessages {
+ schema,
+ messages: polled.messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> {
+ if result == SourceBatchResult::Nack {
+ // Dropping the candidate leaves the live offsets where they were, so the
+ // next poll reads the same rows again. Delivered batches keep their cleanup.
+ let mut pending = self.pending_batch.lock().await;
+ if pending
+ .as_ref()
+ .is_some_and(|pending| !pending.acknowledged && !pending.retiring)
+ {
+ pending.take();
+ }
+ return Ok(());
+ }
+
+ let Some(PendingBatch { mut state, .. }) = self.pending_batch.lock().await.clone() else {
+ return Ok(());
+ };
+
+ let operations = std::mem::take(&mut state.pending_operations);
+ let had_operations = !operations.is_empty();
+ let deadline = tokio::time::Instant::now() + ACK_BATCH_TIMEOUT;
+ let failed_operations = self.apply_pending_operations(operations, deadline).await?;
+
+ if failed_operations.is_empty() {
+ *self.state.lock().await = state.clone();
+ let mut pending = self.pending_batch.lock().await;
+ if had_operations {
+ *pending = Some(PendingBatch {
+ state,
+ acknowledged: false,
+ retiring: true,
+ });
+ } else {
+ pending.take();
+ }
+ } else {
+ state.pending_operations = failed_operations;
+ *self.pending_batch.lock().await = Some(PendingBatch {
+ state,
+ acknowledged: true,
+ retiring: false,
+ });
+ }
+
+ Ok(())
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ let state = restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ pending_operations: Vec::new(),
+ });
+ // The runtime only persists a batch's operations after sending it, so restored
+ // operations belong to a delivered batch and only the cleanup is left to do.
+ let pending_batch = (!state.pending_operations.is_empty()).then(|| PendingBatch {
+ state: state.clone(),
+ acknowledged: true,
+ retiring: false,
+ });
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(state),
+ pending_batch: Mutex::new(pending_batch),
+ consecutive_cleanup_failures: AtomicU32::new(0),
+ verified_cleanup_keys: Mutex::new(HashSet::new()),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ poisoned_tables: Mutex::new(HashSet::new()),
+ tracking_kinds: Mutex::new(HashMap::new()),
+ polls: AtomicU64::new(0),
+ cursor_regressions: Mutex::new(HashMap::new()),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ // `format_offset_value` escapes backslashes, which under NO_BACKSLASH_ESCAPES
+ // would reach MySQL as literal characters and filter on the wrong cursor.
+ .after_connect(|connection, _| {
+ Box::pin(async move {
+ sqlx::query(
+ "SET SESSION sql_mode = REPLACE(@@SESSION.sql_mode, 'NO_BACKSLASH_ESCAPES', '')",
+ )
+ .execute(&mut *connection)
+ .await?;
+ Ok(())
+ })
+ })
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn tracking_column(&self) -> &str {
+ self.config.tracking_column.as_deref().unwrap_or("id")
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn resolve_tracking_columns(&self) -> Result<HashMap<String, OffsetKind>, Error> {
+ // Whether the connector owns the query, and can therefore hold the schema to
+ // what that query will do with it.
+ let strict = self.config.custom_query.is_none();
+ let pool = self.get_pool()?;
+ let tracking_column = self.tracking_column();
+ let mut kinds = HashMap::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ }
+
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ }
+
+ warn!(
+ "tracking_column '{tracking_column}' is not a column of table '{table}'; \
+ assuming custom_query projects it. Its comparison order is resolved from \
+ the first result set instead."
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ // The column's type survives a plain projection, so it is worth rejecting
+ // whatever the query looks like. A custom_query aliasing an expression onto
+ // a name the table also uses is the one false positive, hence the hint.
+ if let Some(reason) = unusable_tracking_type(&data_type) {
+ let aliasing_hint = if self.config.custom_query.is_some() {
+ " If custom_query aliases a different expression to this name, rename the \
+ alias so it does not collide with the column."
+ } else {
+ ""
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': {reason}. Use an integer, timestamp, or string \
+ column.{aliasing_hint}"
+ )));
+ }
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; a row \
+ whose value is NULL cannot produce a cursor and will fail that table's batch. \
+ Have custom_query exclude those rows, or declare the column NOT NULL."
+ );
+ }
+
+ if is_collation_ordered(&data_type) {
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': values must be unique and increase in its collation order \
+ (zero-pad numbers, prefer a _bin collation)"
+ );
+ }
+
+ kinds.insert(table.clone(), offset_kind_for_data_type(&data_type));
+ }
+ Ok(kinds)
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn serialize_pending_state(&self, state: &State) -> Result<ConnectorState, Error> {
+ self.serialize_state(state).ok_or_else(|| {
+ Error::Serialization(format!("failed to serialize {CONNECTOR_NAME} state"))
+ })
+ }
+
+ async fn retiring_state(&self) -> Option<State> {
+ self.pending_batch
+ .lock()
+ .await
+ .as_ref()
+ .filter(|pending| pending.retiring)
+ .map(|pending| pending.state.clone())
+ }
+
+ fn primary_key_column(&self) -> &str {
+ self.config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(self.tracking_column())
+ }
+
+ fn cleanup_target(&self) -> Option<CleanupTarget> {
+ let action = if self.config.delete_after_read.unwrap_or(false) {
+ CleanupAction::Delete
+ } else if let Some(column) = &self.config.processed_column {
+ CleanupAction::MarkProcessed {
+ column: column.clone(),
+ }
+ } else {
+ return None;
+ };
+ Some(CleanupTarget {
+ key_column: self.primary_key_column().to_string(),
+ action,
+ })
+ }
+
+ /// Restored operations were staged under the config of the run that wrote them.
+ /// Running them under a different key or action would touch rows nobody read.
+ fn validate_pending_cleanup(&mut self) -> Result<(), Error> {
+ let configured = self.cleanup_target();
+ for operation in &self.state.get_mut().pending_operations {
+ if configured.as_ref() != Some(&operation.target) {
+ return Err(Error::InitError(format!(
+ "the cleanup configuration changed while cleanup of {} delivered row(s) in \
+ table '{}' is pending; restore primary_key_column '{}' and action {:?}",
+ operation.ids.len(),
+ operation.table,
+ operation.target.key_column,
+ operation.target.action
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ /// `WHERE key IN (...)` on a key that repeats marks or deletes every row sharing a
+ /// value, including ones that were never read. Tables that don't exist yet are
+ /// checked by the first poll that stages cleanup for them.
+ async fn validate_cleanup_keys(&mut self) -> Result<(), Error> {
+ let Some(target) = self.cleanup_target() else {
+ return Ok(());
+ };
+ let pool = self.get_pool()?;
+ let mut verified = HashSet::with_capacity(self.config.tables.len());
+ for table in &self.config.tables {
+ let check = cleanup_key_check(pool, table, &target.key_column)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to check primary_key_column '{}' on table '{table}': {e}",
+ target.key_column
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ verified.insert(table.clone());
+ }
+ CleanupKeyCheck::NotUnique => {
+ return Err(Error::InitError(not_unique_key_message(
+ table,
+ &target.key_column,
+ )));
+ }
+ CleanupKeyCheck::TableMissing => {}
+ }
+ }
+ *self.verified_cleanup_keys.get_mut() = verified;
+ Ok(())
+ }
+
+ async fn cleanup_key_is_unique(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ key_column: &str,
+ ) -> Result<bool, Error> {
+ if self.verified_cleanup_keys.lock().await.contains(table) {
+ return Ok(true);
+ }
+ let check = cleanup_key_check(pool, table, key_column)
+ .await
+ .map_err(|e| {
+ Error::Connection(format!(
+ "failed to check primary_key_column '{key_column}' on table '{table}': {e}"
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ self.verified_cleanup_keys
+ .lock()
+ .await
+ .insert(table.to_string());
+ Ok(true)
+ }
+ CleanupKeyCheck::NotUnique => Ok(false),
+ CleanupKeyCheck::TableMissing => Err(Error::Connection(format!(
+ "table '{table}' disappeared before its primary_key_column could be checked"
+ ))),
+ }
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ /// Best current knowledge of how a table's tracking values compare.
+ async fn tracking_kind(&self, table: &str) -> OffsetKind {
+ self.tracking_kinds
+ .lock()
+ .await
+ .get(table)
+ .copied()
+ .unwrap_or(OffsetKind::Unknown)
+ }
+
+ /// Whether the stored cursor is what limits the rows a poll can see. The built
+ /// polling query always filters on it; a `custom_query` does so only if it
+ /// interpolates `$offset`.
+ fn cursor_gates_query(&self) -> bool {
+ match &self.config.custom_query {
+ Some(query) => query.contains("$offset"),
+ None => true,
+ }
+ }
+
+ /// The offset a poll actually resumes from: the stored cursor, or the configured
+ /// starting point before anything has been stored.
+ fn effective_offset(&self, last_offset: &Option<String>) -> Option<String> {
+ last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let offset_value = self.effective_offset(last_offset).unwrap_or_default();
+ let offset = format_offset_value(&offset_value, kind);
+ let quoted_table = quote_qualified_identifier(table)?;
+ let limit = batch_size.to_string();
+ let now = Utc::now();
+ let now_unix = now.timestamp().to_string();
+ let now = now.to_rfc3339();
+ let replacements = [
+ ("$table", quoted_table.as_str()),
+ ("$offset", offset.as_str()),
+ ("$limit", limit.as_str()),
+ ("$now_unix", now_unix.as_str()),
+ ("$now", now.as_str()),
+ ];
+
+ Ok(substitute_query_tokens(query, &replacements))
+ }
+
+ /// The query is substituted as text and re-executed once per configured table on
+ /// every poll for the life of the process, so a shape that cannot terminate does
+ /// not fail, it repeats. What can be judged from the text alone is judged here,
+ /// at startup, rather than left to the first poll.
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let tracking_column = self.tracking_column();
+
+ match leading_statement_token(query) {
+ None => {
+ return Err(Error::InitError(
+ "custom_query has no statement: it is empty, or nothing but comments"
+ .to_string(),
+ ));
+ }
+ Some(token) if !READ_STATEMENT_STARTS.contains(&token.as_str()) => {
+ return Err(Error::InitError(format!(
+ "custom_query must be a read starting with SELECT, WITH or TABLE, found \
+ '{token}'"
+ )));
+ }
+ Some(_) => {}
+ }
+
+ // Collapsed so `ORDER\nBY` and `ORDER BY` read the same as `ORDER BY`; the
+ // check hard-fails, so a formatting difference must not decide it.
+ let normalized = query
+ .split_whitespace()
+ .collect::<Vec<_>>()
+ .join(" ")
+ .to_uppercase();
+
+ if query.contains("$offset") {
+ if !normalized.contains("ORDER BY") {
+ return Err(Error::InitError(format!(
+ "custom_query filters on $offset but has no ORDER BY, add \
+ `ORDER BY {tracking_column}`"
+ )));
+ }
+ } else if !self.config.delete_after_read.unwrap_or(false)
+ && self.config.processed_column.is_none()
+ {
+ return Err(Error::InitError(format!(
+ "custom_query must filter with $offset (`WHERE {tracking_column} > $offset`), or \
+ set delete_after_read or processed_column"
+ )));
+ }
+
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "custom_query uses $table but no tables are configured".to_string(),
+ ));
+ }
+ // Without $table the same SQL is executed once per table, so identical rows
+ // would be emitted repeatedly, each carrying a different table's metadata,
+ // message ID, and tracking offset. Table identity is not recoverable from a
+ // static query, so reject the combination instead of guessing one.
+ if !query.contains("$table") && self.config.tables.len() > 1 {
+ return Err(Error::InitError(format!(
+ "custom_query has no $table placeholder but {} tables are configured ({}), use \
+ $table in the query or configure a single table",
+ self.config.tables.len(),
+ self.config.tables.join(", ")
+ )));
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = self.effective_offset(last_offset) {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(&offset, kind)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn apply_pending_operations(
+ &self,
+ operations: Vec<PendingOperation>,
+ deadline: tokio::time::Instant,
+ ) -> Result<Vec<PendingOperation>, Error> {
+ let mut failed_operations = Vec::new();
+ let mut operations = operations.into_iter();
+ while let Some(operation) = operations.next() {
+ if tokio::time::Instant::now() >= deadline {
+ failed_operations.push(operation);
+ failed_operations.extend(operations);
+ break;
+ }
+ let result = tokio::time::timeout_at(deadline, async {
+ let pool = self.get_pool()?;
+ self.mark_or_delete_processed_rows(pool, &operation).await
+ })
+ .await;
+ match result {
+ Ok(Ok(())) => {
+ self.consecutive_cleanup_failures
+ .store(0, Ordering::Relaxed);
+ }
+ Ok(Err(error)) => {
+ self.record_cleanup_failure(&operation, &error)?;
+ failed_operations.push(operation);
+ }
+ Err(_) => {
+ self.record_cleanup_failure(
+ &operation,
+ "cleanup exceeded the 10s budget for one Ack",
+ )?;
+ failed_operations.push(operation);
+ }
+ }
+ }
+ Ok(failed_operations)
+ }
+
+ fn record_cleanup_failure(
+ &self,
+ operation: &PendingOperation,
+ error: impl std::fmt::Display,
+ ) -> Result<(), Error> {
+ let consecutive_failures = self
+ .consecutive_cleanup_failures
+ .fetch_add(1, Ordering::Relaxed)
+ .saturating_add(1);
+ error!(
+ "Failed to mark or delete {} delivered row(s) in table '{}' for {CONNECTOR_NAME} \
+ connector ID: {}, retrying on the next poll ({consecutive_failures} consecutive \
+ failures): {error}",
+ operation.ids.len(),
+ operation.table,
+ self.id
+ );
+ if consecutive_failures >= MAX_CONSECUTIVE_CLEANUP_FAILURES {
+ return Err(Error::Connection(format!(
+ "stopping {CONNECTOR_NAME} connector ID {} after {consecutive_failures} \
+ consecutive mark/delete failures; the cleanup stays in the saved state and \
+ runs again on restart",
+ self.id
+ )));
+ }
+ Ok(())
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ operation: &PendingOperation,
+ ) -> Result<(), Error> {
+ let PendingOperation { table, ids, target } = operation;
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(&target.key_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = match &target.action {
+ CleanupAction::Delete => {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
Review Comment:
fixed.
Instead of saving a tracking value for every row, each pending cleanup now saves the batch's highest tracking value, and the cleanup only touches rows at or below it: DELETE FROM t WHERE pk IN (...) AND tracking_column <= <batch max>
A replacement row always gets a newer tracking value, so the cleanup leaves it alone. This also avoids exact-equality matching on DATETIME and DECIMAL values, and the state stays small.
Since this needs the tracking column to exist in the table, a computed or aliased tracking column is now rejected when delete_after_read or processed_column is set. The README covers it.
Added integration tests that replay a pending cleanup after a key is reused, for both the built-in query and a custom_query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3484345958
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1358 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ if self.config.delete_after_read.unwrap_or(false) {
+ let delete_query =
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})");
+
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+
+ sqlx::query(sqlx::AssertSqlSafe(delete_query))
+ .execute(pool)
+ .await
+ .map_err(|e| {
+ error!("Failed to delete processed rows: {e}");
+ Error::InvalidRecord
+ })?;
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ let update_query = format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ );
+
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+
+ sqlx::query(sqlx::AssertSqlSafe(update_query))
+ .execute(pool)
+ .await
+ .map_err(|e| {
+ error!("Failed to mark rows as processed: {e}");
+ Error::InvalidRecord
+ })?;
+ }
+
+ Ok(())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+ let mut messages = Vec::new();
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Collect state updates to apply after processing
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for table in &self.config.tables {
+ let table_config = RowProcessingConfig {
+ table,
+ ..row_config
+ };
+
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.validate_custom_query(custom_query)?;
+ self.substitute_query_params(custom_query, table, &last_offset, batch_size)
+ } else {
+ self.build_polling_query(table, tracking_column, &last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut max_offset: Option<String> = None;
+ let mut processed_ids: Vec<String> = Vec::new();
+
+ let mut count_per_table = 0;
+ for row in rows {
+ let processed = self.process_row(&row, &table_config)?;
+
+ if let Some(pk) = processed.row_pk {
+ processed_ids.push(pk);
+ }
+ if let Some(offset) = processed.max_offset {
+ max_offset = Some(offset);
+ }
+
+ messages.push(processed.message);
+ count_per_table += 1;
+ total_processed += 1;
+ }
+
+ // Database I/O without holding the lock
+ if !processed_ids.is_empty() {
+ self.mark_or_delete_processed_rows(pool, table, pk_column, &processed_ids)
+ .await?;
Review Comment:
Reworked poll_tables into two phases with per-table isolation:
Phase 1 — fetch + process, no side effects. Each table builds its own `TableBatch`. On failure (decode error, bad query), that table is logged and skipped for the cycle — its rows stay in MySQL and retry next poll. Other tables keep going.
Phase 2 — commit, per-table. `mark_or_delete` runs per table; only on success are messages appended and the offset advanced. Failures log and continue — one table's mark failure can't discard another's already-committed messages. Also added retry for transient errors (deadlock/lock-wait/connection drop) via `with_retry`.
Two things this doesn't cover -
Mark still happens before `producer.send`, so a send failure can still drop a batch. That needs a post-send ack on the Source trait — same gap exists in postgres_source, so I'd rather track it as a separate issue. Happy to open one if you agree.
With per-table skip, poll() returns Ok, so a stuck table only surfaces in the error log. Open to adding a metric if you'd prefer it more visible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on PR #3568:
URL: https://github.com/apache/iggy/pull/3568#issuecomment-4811003600
Hi @atharvalade, thanks for the detailed feedback!
On the process side — I did reach out on the Discord community before starting, where I got clearance to proceed with the connector work. I understand that might not substitute for a formal maintainer approval comment on the issue itself, and I'll make sure to get that explicitly documented going forward before picking up anything in high-risk areas.
On the CI failures — on it, will get those green shortly.
Also working through the inline review comments now. Will push the fixes once the CI is sorted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3554133765
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).execute(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ batches.push(batch);
+ }
+ Err(error) => {
Review Comment:
per-table fetch failure logged+skipped; poll() returns Ok partial batch. Intentional vs postgres abort. **Fix**: document or error metric.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r4187740168
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,3317 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some(
+ "at most 256 distinct values cannot be unique per row, and at display width 1 MySQL \
+ spells the type BOOLEAN, which reaches the connector as true/false with no ordered \
+ scalar form",
+ );
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some(
+ "binary values are carried as base64 text, which does not order the way the raw \
+ bytes do, and that text is what gets compared against the column",
+ );
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some(
+ "MySQL orders these by declaration index while only their labels reach the \
+ connector, and a bounded set of labels cannot be unique per row either",
+ );
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]. Every row's cursor \
+ comes from that column, so the same rows would be re-published on every poll while the \
+ stored offset never moved.",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL, or \
+ of a type with no ordered scalar form (tinyint(1), JSON). The cursor is taken from that \
+ value, so publishing the row would advance nothing and the same rows would be re-read on \
+ every poll."
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "a row arrived below the offset the poll resumed from by tracking_column \
+ '{column}': stored offset '{previous}', first row '{value}'. Taking the \
+ cursor from the last row would move it backwards. Filter the query on \
+ '{column}' with $offset so a batch cannot reach behind its own cursor."
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. The cursor is taken from the last row, so it would \
+ move backwards and rows would be skipped permanently. Order the query \
+ ascending by '{column}'."
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ // Every poll iterates `tables`, so an empty list makes the connector a
+ // silent no-op rather than an obvious misconfiguration.
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty. Even when custom_query hardcodes the table, list it \
+ here: the table name supplies the offset key, the deterministic message ID, \
+ and the table_name metadata field"
+ .to_string(),
+ ));
+ }
+
+ // A zero batch would make every fetch look like a full batch, so `poll`
+ // would drop its pacing sleep and spin on `LIMIT 0` queries forever.
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ // Only when the operator owns the ORDER BY. The built query sorts ascending by
+ // construction, so a one-row batch is safe there and legitimate for a slow table.
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset: a \
+ one-row batch offers no pair to compare, so the ordering check reduces to \
+ the row against the offset it resumed from, and a query ordered descending \
+ opens above that offset on every poll. The cursor would walk to the table's \
+ maximum and skip every row below it with nothing reporting it"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "the connector does not parse custom_query, so it has to be told which projected \
+ column supplies the cursor: name the one the query orders by and filters with \
+ $offset"
+ } else {
+ "it has to be unique and monotonically increasing under the order MySQL sorts it \
+ in."
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set: every row's cursor is taken from it, and {guidance}."
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ // Skip the pacing delay while draining a backlog
+ if !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ // Idle cycles produce no messages and leave offsets/processed_rows
+ // untouched, so return None to let the runtime skip the state fsync.
+ let persisted_state = if messages.is_empty() {
+ None
+ } else {
+ self.serialize_state(&state)
+ };
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ poisoned_tables: Mutex::new(HashSet::new()),
+ tracking_kinds: Mutex::new(HashMap::new()),
+ polls: AtomicU64::new(0),
+ cursor_regressions: Mutex::new(HashMap::new()),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ /// The column every cursor is read from. `open()` refuses a config that leaves it
+ /// unset, so the fallback is unreachable in a running connector.
+ fn tracking_column(&self) -> &str {
+ self.config.tracking_column.as_deref().unwrap_or("id")
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn resolve_tracking_columns(&self) -> Result<HashMap<String, OffsetKind>, Error> {
+ // Whether the connector owns the query, and can therefore hold the schema to
+ // what that query will do with it.
+ let strict = self.config.custom_query.is_none();
+ let pool = self.get_pool()?;
+ let tracking_column = self.tracking_column();
+ let mut kinds = HashMap::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ }
+
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ }
+
+ warn!(
+ "tracking_column '{tracking_column}' is not a column of table '{table}'; \
+ assuming custom_query projects it. Its comparison order is resolved from \
+ the first result set instead."
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ // The column's type survives a plain projection, so it is worth rejecting
+ // whatever the query looks like. A custom_query aliasing an expression onto
+ // a name the table also uses is the one false positive, hence the hint.
+ if let Some(reason) = unusable_tracking_type(&data_type) {
+ let aliasing_hint = if self.config.custom_query.is_some() {
+ " If custom_query aliases a different expression to this name, rename the \
+ alias so it does not collide with the column."
+ } else {
+ ""
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': {reason}. Use an integer, timestamp, or string \
+ column.{aliasing_hint}"
+ )));
+ }
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; a row \
+ whose value is NULL cannot produce a cursor and will fail that table's batch. \
+ Have custom_query exclude those rows, or declare the column NOT NULL."
+ );
+ }
+
+ if is_collation_ordered(&data_type) {
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}', so its cursor is compared in the column's collation and has \
+ to be unique and monotonically increasing in that order. Unpadded digit \
+ strings are not: '17' sorts below '9', after which the connector idles at \
+ the collation maximum forever. Zero-padded numbers, ULIDs and KSUIDs are. A \
+ case- or accent-insensitive collation such as the default utf8mb4_0900_ai_ci \
+ also breaks uniqueness, since 'ORD-1a' equals 'ORD-1A' and one of the two is \
+ dropped, so prefer a _bin collation."
+ );
+ }
+
+ kinds.insert(table.clone(), offset_kind_for_data_type(&data_type));
+ }
+ Ok(kinds)
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ /// Best current knowledge of how a table's tracking values compare.
+ async fn tracking_kind(&self, table: &str) -> OffsetKind {
+ self.tracking_kinds
+ .lock()
+ .await
+ .get(table)
+ .copied()
+ .unwrap_or(OffsetKind::Unknown)
+ }
+
+ /// Whether the stored cursor is what limits the rows a poll can see. The built
+ /// polling query always filters on it; a `custom_query` does so only if it
+ /// interpolates `$offset`.
+ fn cursor_gates_query(&self) -> bool {
+ match &self.config.custom_query {
+ Some(query) => query.contains("$offset"),
+ None => true,
+ }
+ }
+
+ /// The offset a poll actually resumes from: the stored cursor, or the configured
+ /// starting point before anything has been stored.
+ fn effective_offset(&self, last_offset: &Option<String>) -> Option<String> {
+ last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let offset_value = self.effective_offset(last_offset).unwrap_or_default();
+ let offset = format_offset_value(&offset_value, kind);
+ let now = Utc::now();
+
+ let q = query
+ .replace("$table", "e_qualified_identifier(table)?)
+ .replace("$offset", &offset)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now_unix", &now.timestamp().to_string())
+ .replace("$now", &now.to_rfc3339());
+
+ Ok(q)
+ }
+
+ /// The query is substituted as text and re-executed once per configured table on
+ /// every poll for the life of the process, so a shape that cannot terminate does
+ /// not fail, it repeats. What can be judged from the text alone is judged here,
+ /// at startup, rather than left to the first poll.
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let tracking_column = self.tracking_column();
+
+ match leading_statement_token(query) {
+ None => {
+ return Err(Error::InitError(
+ "custom_query has no statement: it is empty, or nothing but comments"
+ .to_string(),
+ ));
+ }
+ Some(token) if !READ_STATEMENT_STARTS.contains(&token.as_str()) => {
+ return Err(Error::InitError(format!(
+ "custom_query must be a read, but it starts with '{token}'. It is executed \
+ once per configured table on every poll for the life of the process, so a \
+ statement that writes would re-run forever. Start it with SELECT, WITH or \
+ TABLE."
+ )));
+ }
+ Some(_) => {}
+ }
+
+ // Collapsed so `ORDER\nBY` and `ORDER BY` read the same as `ORDER BY`; the
+ // check hard-fails, so a formatting difference must not decide it.
+ let normalized = query
+ .split_whitespace()
+ .collect::<Vec<_>>()
+ .join(" ")
+ .to_uppercase();
+
+ // Two shapes, two failures. With `$offset` the stored cursor is what limits
+ // the next poll, and it is only the batch's maximum when the rows arrive
+ // ascending, so an unordered query skips rows. Without `$offset` nothing
+ // limits the poll at all: the cursor is written but never read back, so the
+ // same rows return forever unless the rows leave the result set. Only
+ // `delete_after_read` guarantees that on its own - `processed_column` marks
+ // rows the connector never filters on outside the query it builds itself, so
+ // that arm rests on the operator's WHERE and is the weaker of the two.
+ if query.contains("$offset") {
+ if !normalized.contains("ORDER BY") {
+ return Err(Error::InitError(format!(
+ "custom_query filters on $offset but has no ORDER BY. The next offset is the \
+ tracking value of the last row returned, which is only the highest one when \
+ the batch arrives ascending by '{tracking_column}'; without that, rows below \
+ it are skipped permanently. Add `ORDER BY {tracking_column}`."
+ )));
+ }
+ } else if !self.config.delete_after_read.unwrap_or(false)
+ && self.config.processed_column.is_none()
+ {
+ return Err(Error::InitError(format!(
+ "custom_query does not filter on $offset, so every poll returns the same rows and \
+ publishes them again. Interpolate $offset (`WHERE {tracking_column} > $offset`, \
+ which also needs an ORDER BY), or take the rows out of the next poll by setting \
+ delete_after_read, which deletes them, or processed_column, which marks them - \
+ with processed_column the query has to exclude the marked rows itself, since the \
+ connector only appends that filter to the query it builds."
+ )));
+ }
+
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "custom_query uses $table but no tables are configured, so the placeholder \
+ can never resolve; list the tables to poll in `tables`"
+ .to_string(),
+ ));
+ }
+ // Without $table the same SQL is executed once per table, so identical rows
+ // would be emitted repeatedly, each carrying a different table's metadata,
+ // message ID, and tracking offset. Table identity is not recoverable from a
+ // static query, so reject the combination instead of guessing one.
+ if !query.contains("$table") && self.config.tables.len() > 1 {
+ return Err(Error::InitError(format!(
+ "custom_query has no $table placeholder but {} tables are configured ({}); \
+ the same query would run once per table and emit its rows multiple times. \
+ Use $table in the query, or configure a single table.",
+ self.config.tables.len(),
+ self.config.tables.join(", ")
+ )));
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = self.effective_offset(last_offset) {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(&offset, kind)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[PkValue],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({placeholders})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ // The query and its binds are rebuilt on every attempt because with_retry
+ // calls the closure once per try and each `execute` consumes the query.
+ with_retry(
+ || {
+ let mut prepared = sqlx::query(sqlx::AssertSqlSafe(query.as_str()));
+ for id in ids {
+ prepared = match id {
+ PkValue::Text(text) => prepared.bind(text),
+ PkValue::Bytes(bytes) => prepared.bind(bytes),
+ };
+ }
+ prepared.execute(pool)
+ },
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.tracking_column();
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ //
+ // An ordering violation is handled one step further. Retrying it would
+ // re-run the same query, re-decode the same rows, and hit the same failure
+ // forever, so the table is latched off instead of being retried each cycle.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ let poll_number = self.polls.fetch_add(1, Ordering::Relaxed) + 1;
+ let poisoned = { self.poisoned_tables.lock().await.clone() };
+ let mut newly_poisoned: Vec<String> = Vec::new();
+
+ for table in &self.config.tables {
+ if poisoned.contains(table) {
+ continue;
+ }
+
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ if !batch.messages.is_empty() {
+ self.cursor_regressions.lock().await.remove(table);
+ }
+ // A full batch means the LIMIT was hit, so more rows are waiting.
+ any_batch_full |= batch.messages.len() as u32 >= batch_size;
+ batches.push(batch);
+ }
+ Err(FetchError::Ordering(reason) | FetchError::MissingTrackingColumn(reason)) => {
+ let remedy = if self.config.custom_query.is_some() {
+ "with a corrected query"
+ } else {
+ "with a corrected tracking_column, since the connector wrote this query \
+ itself"
+ };
+ error!(
+ "Table '{table}' at offset {}: {reason} No rows were published and the \
+ offset was not advanced. This is deterministic, so the table is disabled \
+ until the connector is restarted {remedy}.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ newly_poisoned.push(table.clone());
+ }
+ Err(FetchError::CursorRegression(reason)) => {
+ // Cleared only by a batch that carried rows, so the run counts polls
+ // since the table last published anything. An idle poll produces as
+ // little as a rejected one does and must not read as recovery.
+ let consecutive = {
+ let mut regressions = self.cursor_regressions.lock().await;
+ let consecutive = regressions.entry(table.clone()).or_insert(0);
+ *consecutive += 1;
+ *consecutive
+ };
+ let offset = last_offset.as_deref().unwrap_or("<start>");
+ if consecutive >= CURSOR_REGRESSION_ERROR_THRESHOLD {
+ error!(
+ "Table '{table}' at offset {offset}: {reason} No rows were published \
+ and the offset was not advanced. This is the {consecutive}th \
+ consecutive poll to end this way, so it is no longer resolving on \
+ its own: the table stays enabled but has published nothing since."
+ );
+ } else {
+ warn!(
+ "Table '{table}' at offset {offset}: {reason} No rows were published \
+ and the offset was not advanced. The table stays enabled, since this \
+ can resolve without a config change."
+ );
+ }
+ }
+ Err(FetchError::StaleOffsetKind(reason)) => {
+ warn!(
+ "Table '{table}' at offset {}: {reason} No rows were published and the \
+ offset was not advanced. The next poll rebuilds the query from the type \
+ the result set reported.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::UnusableTrackingValue(reason)) => {
+ error!(
+ "Table '{table}' at offset {}: {reason} No rows were published and the \
+ offset was not advanced. The table stays enabled so a corrected row is \
+ picked up without a restart, but nothing will be published until then - \
+ this repeats every poll.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::Other(error)) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ if !newly_poisoned.is_empty() {
+ let mut poisoned_tables = self.poisoned_tables.lock().await;
+ poisoned_tables.extend(newly_poisoned);
+ if self
+ .config
+ .tables
+ .iter()
+ .all(|table| poisoned_tables.contains(table))
+ {
+ error!(
+ "All configured tables are disabled because their batches cannot yield a \
+ cursor; {CONNECTOR_NAME} connector with ID: {} is idle until restarted",
+ self.id
+ );
+ }
+ }
+
+ // From the next poll on a disabled table is skipped before anything is logged,
+ // so the one line written when it was disabled can scroll out of the operator's
+ // log window and leave a dark connector looking healthy.
+ if poll_number.is_multiple_of(DISABLED_TABLES_LOG_INTERVAL) {
+ let poisoned_tables = self.poisoned_tables.lock().await;
+ if !poisoned_tables.is_empty() {
+ let mut disabled: Vec<&str> = poisoned_tables.iter().map(String::as_str).collect();
+ disabled.sort_unstable();
+ error!(
+ "{} of {} configured table(s) are still disabled and are not being polled by \
+ {CONNECTOR_NAME} connector with ID: {}, and will stay that way until it is \
+ restarted: {}",
+ disabled.len(),
+ self.config.tables.len(),
+ self.id,
+ disabled.join(", ")
+ );
+ }
+ }
+
+ // Phase 2: commit each table independently. If the mark/delete call fails
+ // for a table we just skip its messages this cycle instead of queuing them
+ // - `mark_or_delete_processed_rows` already retries transient errors, so a
+ // failure here is permanent and the table's rows stay in MySQL to be
+ // retried next cycle, without blocking the others.
+ //
+ // Doesn't cover the case where mark/delete succeeds but the runtime never
+ // gets to publish the batch (crash, send failure, etc) - those rows are
+ // gone from MySQL with nothing delivered.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ let mark_or_delete = self.config.delete_after_read.unwrap_or(false)
+ || self.config.processed_column.is_some();
+
+ for mut batch in batches {
+ if mark_or_delete && !batch.messages.is_empty() && batch.processed_ids.is_empty() {
+ error!(
+ "Table '{}': mark/delete is configured but no primary keys were extracted \
+ from {} row(s), skipping this cycle so rows are not published without being \
+ marked or deleted (check that '{pk_column}' is projected and scalar)",
+ batch.table,
+ batch.messages.len()
+ );
+ continue;
+ }
+
+ if !batch.processed_ids.is_empty()
+ && let Err(error) = self
+ .mark_or_delete_processed_rows(
+ pool,
+ &batch.table,
+ pk_column,
+ &batch.processed_ids,
+ )
+ .await
+ {
+ error!(
+ "Failed to mark or delete processed rows for table '{}', skipping this cycle: {error}",
+ batch.table
+ );
+ continue;
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ state_updates.push((batch.table, offset));
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
+ }
+ }
+
+ self.last_batch_full
+ .store(any_batch_full, Ordering::Relaxed);
+ Ok(messages)
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, FetchError> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query_kind = self.tracking_kind(table).await;
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.substitute_query_params(custom_query, table, last_offset, batch_size, query_kind)?
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size, query_kind)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ // Settled once for the whole batch rather than per row, since the column list
+ // is a property of the result set.
+ if let Some(first) = rows.first() {
+ ensure_tracking_column_projected(
+ first.columns().iter().map(|column| column.name()),
+ tracking_column,
+ )?;
+ }
+
+ let resume_from = if self.cursor_gates_query() {
+ self.effective_offset(last_offset)
+ } else {
+ None
+ };
+
+ // The result set describes what this query actually returned, so it outranks
+ // whatever `information_schema` said at startup and is the only answer
+ // available for a computed column.
+ let kind = match rows.first().and_then(|row| {
+ observed_tracking_kind(row, tracking_column).filter(|observed| *observed != query_kind)
+ }) {
+ Some(observed) => {
+ self.tracking_kinds
+ .lock()
+ .await
+ .insert(table.to_string(), observed);
+ // The query was already sent with the stale kind, so its offset literal
+ // may be quoted for the wrong type.
+ if let Some(offset) = &resume_from
+ && offset_literal_diverged(offset, query_kind, observed)
+ {
+ return Err(FetchError::StaleOffsetKind(format!(
+ "the poll filtered on '{tracking_column}' with {}, written for the \
+ comparison order known before this poll, but the result set reports an \
+ order whose literal is {}. A filter that disagrees with the query's own \
+ ordering strands rows below the cursor permanently, so the batch was \
+ discarded.",
+ format_offset_value(offset, query_kind),
+ format_offset_value(offset, observed)
+ )));
+ }
+ observed
+ }
+ None => query_kind,
+ };
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ let mut ordering = OrderingGuard::new(tracking_column, kind, resume_from);
+ for row in rows {
+ let processed = self.process_row(&row, &table_config).map_err(|e| {
+ error!(
+ "Failed to decode row in table '{table}' (pk {}): {e}",
+ pk_for_log(&row, table_config.pk_column)
+ );
+ e
+ })?;
+
+ let Some(value) = processed.tracking_value else {
+ return Err(unusable_tracking_value(
+ tracking_column,
+ &pk_for_log(&row, table_config.pk_column),
+ ));
+ };
+ ordering
+ .accept(value)
+ .map_err(|violation| match violation {
+ OrderingViolation::WithinBatch(reason) => FetchError::Ordering(reason),
+ OrderingViolation::BelowCursor(reason) => FetchError::CursorRegression(reason),
+ })?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+
+ batch.messages.push(processed.message);
+ }
+ batch.max_offset = ordering.into_cursor();
+
+ Ok(batch)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<PkValue> = None;
+ let mut tracking_value: Option<String> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
+ let mut extracted_payload: Option<Vec<u8>> = None;
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ if name.eq_ignore_ascii_case(config.payload_col) {
+ extracted_payload =
+ Some(self.extract_payload_column(row, i, config.payload_format)?);
+ }
+ if name.eq_ignore_ascii_case(config.tracking_column) {
+ let value = extract_column_value(row, i)?;
+ tracking_value = column_value_as_string(row, i, &value)?;
+ }
+ if name.eq_ignore_ascii_case(config.pk_column) {
+ row_pk = extract_pk_value(row, i)?;
+ }
+ }
+
+ // Every extract_payload_column arm yields Ok for a NULL column, so None
+ // here means the column was absent. Falling back to the whole-row JSON
+ // would publish bytes contradicting the schema derived from payload_format.
+ let payload = extracted_payload.ok_or_else(|| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': payload_column '{}' not present in the result set",
+ config.table, config.payload_col
+ ))
+ })?;
+
+ return Ok(build_processed_row(
+ config.table,
+ payload,
+ tracking_value,
+ row_pk,
+ ));
+ }
+
+ let mut data = serde_json::Map::new();
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ let column_name = if config.snake_case_columns {
+ to_snake_case(name)
+ } else {
+ name.to_string()
+ };
+ let value = extract_column_value(row, i)?;
+ if name.eq_ignore_ascii_case(config.tracking_column) {
+ tracking_value = column_value_as_string(row, i, &value)?;
+ }
+ if name.eq_ignore_ascii_case(config.pk_column) {
+ row_pk = extract_pk_value(row, i)?;
+ }
+ data.insert(column_name, value);
+ }
+
+ let payload = if config.include_metadata {
+ let record = DatabaseRecord {
+ table_name: config.table.to_string(),
+ operation_type: "SELECT".to_string(),
+ timestamp: Utc::now(),
+ data: serde_json::Value::Object(data),
+ old_data: None,
+ };
+ simd_json::to_vec(&record).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': failed to serialize row to JSON: {e}",
+ config.table
+ ))
+ })?
+ } else {
+ simd_json::to_vec(&data).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': failed to serialize row to JSON: {e}",
+ config.table
+ ))
+ })?
+ };
+
+ Ok(build_processed_row(
+ config.table,
+ payload,
+ tracking_value,
+ row_pk,
+ ))
+ }
+}
+
+/// Type-faithful primary-key extraction. Binary columns are taken as raw bytes
+/// rather than routed through the base64 path in `extract_column_value`, so the
+/// value can be bound back into `WHERE pk IN (...)` and match the row. Everything
+/// else takes the string form MySQL parses back to the same value. Returns None
+/// only for a NULL pk, which a real primary key cannot be.
+fn extract_pk_value(row: &MySqlRow, column_index: usize) -> Result<Option<PkValue>, Error> {
+ let column = &row.columns()[column_index];
+ let type_name = column.type_info().name();
+ match type_name {
+ "BINARY" | "VARBINARY" | "TINYBLOB" | "BLOB" | "MEDIUMBLOB" | "LONGBLOB" => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "primary key column '{}' (MySQL type '{type_name}'): {e}",
+ column.name()
+ ))
+ })?;
+ Ok(bytes.map(PkValue::Bytes))
+ }
+ _ => {
+ let value = extract_column_value(row, column_index)?;
+ Ok(column_value_as_string(row, column_index, &value)?.map(PkValue::Text))
+ }
+ }
+}
+
+fn extract_column_value(row: &MySqlRow, column_index: usize) -> Result<serde_json::Value, Error> {
+ let column = &row.columns()[column_index];
+ let type_name = column.type_info().name();
+
+ let to_err = |e: sqlx::Error| {
+ Error::InvalidRecordValue(format!(
+ "column '{}' (MySQL type '{type_name}'): {e}",
+ column.name()
+ ))
+ };
+
+ match type_name {
+ "BOOLEAN" => {
+ let value: Option<bool> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::Bool)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT" => {
+ let value: Option<i8> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT" => {
+ let value: Option<i16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT" | "INT" => {
+ let value: Option<i32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT" => {
+ let value: Option<i64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT UNSIGNED" => {
+ let value: Option<u8> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT UNSIGNED" => {
+ let value: Option<u16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT UNSIGNED" | "INT UNSIGNED" => {
+ let value: Option<u32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT UNSIGNED" => {
+ let value: Option<u64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "FLOAT" => {
+ let value: Option<f32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as f64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DOUBLE" => {
+ let value: Option<f64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DECIMAL" => {
+ let value: Option<rust_decimal::Decimal> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|d| serde_json::Value::String(d.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIT" => {
+ let value: Option<u64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "YEAR" => {
+ let value: Option<u16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATE" => {
+ let value: Option<NaiveDate> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|d| serde_json::Value::String(d.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TIME" => {
+ let value: Option<NaiveTime> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|t| serde_json::Value::String(t.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATETIME" => {
+ let value: Option<NaiveDateTime> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|dt| serde_json::Value::String(dt.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TIMESTAMP" => {
+ let value: Option<DateTime<Utc>> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|dt| serde_json::Value::String(dt.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT" | "MEDIUMTEXT" | "LONGTEXT" | "ENUM" | "SET" => {
+ let value: Option<String> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BINARY" | "VARBINARY" | "TINYBLOB" | "BLOB" | "MEDIUMBLOB" | "LONGBLOB" => {
+ let value: Option<Vec<u8>> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|bytes| {
+ serde_json::Value::String(
+ base64::engine::general_purpose::STANDARD.encode(&bytes),
+ )
+ })
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "JSON" => {
+ let value: Option<serde_json::Value> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value.unwrap_or(serde_json::Value::Null))
+ }
+ "NULL" => Ok(serde_json::Value::Null),
+ _ => {
+ let column_name = column.name();
+ warn!(
+ "Column '{column_name}' has unrecognized MySQL type '{type_name}', \
+ attempting text extraction"
+ );
+ if let Ok(text) = row.try_get::<Option<String>, _>(column_index) {
+ return Ok(text
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null));
+ }
+ if let Ok(bytes) = row.try_get::<Option<Vec<u8>>, _>(column_index) {
+ return Ok(bytes
+ .map(|b| {
+ serde_json::Value::String(
+ base64::engine::general_purpose::STANDARD.encode(&b),
+ )
+ })
+ .unwrap_or(serde_json::Value::Null));
+ }
+ error!(
+ "Column '{column_name}' has unsupported MySQL type '{type_name}', \
+ returning null"
+ );
+ Ok(serde_json::Value::Null)
+ }
+ }
+}
+
+/// Best-effort primary-key value for diagnostics when a row fails to decode.
+/// Falls back to "unknown" if the pk column is absent or itself fails to extract,
+/// so logging a decode failure never masks it with a second error.
+fn pk_for_log(row: &MySqlRow, pk_column: &str) -> String {
+ row.columns()
+ .iter()
+ .position(|column| column.name().eq_ignore_ascii_case(pk_column))
+ .and_then(|index| extract_column_value(row, index).ok())
+ .and_then(|value| value_as_string(&value))
+ .unwrap_or_else(|| "unknown".to_string())
+}
+
+fn value_as_string(value: &serde_json::Value) -> Option<String> {
+ match value {
+ serde_json::Value::String(s) => Some(s.clone()),
+ serde_json::Value::Number(n) => Some(n.to_string()),
+ _ => None,
+ }
+}
+
+/// The string form of a column value that MySQL parses back to the value it came
+/// from, which for `TIMESTAMP` is not its payload form: that one carries chrono's
+/// trailing ` UTC`. A `SELECT` only warns about the superfluous delimiter, but the
+/// same conversion inside the mark/delete DML is an error under the default
+/// `STRICT_TRANS_TABLES`, so both the cursor and the `WHERE pk IN (...)` binding
+/// take this form. Only the payload keeps chrono's rendering.
+fn column_value_as_string(
+ row: &MySqlRow,
+ column_index: usize,
+ value: &serde_json::Value,
+) -> Result<Option<String>, Error> {
+ let column = &row.columns()[column_index];
+ if column.type_info().name() != "TIMESTAMP" {
+ return Ok(value_as_string(value));
+ }
+
+ let timestamp: Option<DateTime<Utc>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "tracking column '{}' (MySQL type 'TIMESTAMP'): {e}",
+ column.name()
+ ))
+ })?;
+ Ok(timestamp.map(timestamp_cursor))
+}
+
+/// Fixed width, so the collation comparison a temporal cursor is ordered by agrees
+/// with the instants the values stand for.
+fn timestamp_cursor(timestamp: DateTime<Utc>) -> String {
+ timestamp.format("%Y-%m-%d %H:%M:%S%.6f").to_string()
+}
+
+fn build_processed_row(
+ table: &str,
+ payload: Vec<u8>,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+) -> ProcessedRow {
+ let now = Utc::now().timestamp_micros() as u64;
+ // The key is the pk's string form, so changing how a column type renders re-keys
+ // every row of that type. Once connectors are in the field that breaks dedup for
+ // any row replayed across the change, so weigh it before touching
+ // `column_value_as_string`.
+ let pk_key = row_pk.as_ref().map(PkValue::as_key);
+ let id = message_id(table, pk_key.as_deref().or(tracking_value.as_deref()));
+ ProcessedRow {
+ message: ProducedMessage {
+ id: Some(id),
+ headers: None,
+ checksum: None,
+ timestamp: Some(now),
+ origin_timestamp: Some(now),
+ payload,
+ },
+ tracking_value,
+ row_pk,
+ }
+}
+
+/// Deterministic message id so a row replayed after a restart keeps the same id
+/// and downstream can dedup it. Derived from the table plus the row's stable key
+/// (its primary key, or the unique/monotonic tracking value). The random branch
+/// is unreachable for a contract-compliant tracking column - it only avoids a
+/// panic if a row yields no usable key, at the cost of that one row not being
+/// idempotent on replay.
+fn message_id(table: &str, key: Option<&str>) -> u128 {
+ match key {
+ Some(key) => {
+ let name = format!("{table}\0{key}");
+ Uuid::new_v5(&MESSAGE_ID_NAMESPACE, name.as_bytes()).as_u128()
+ }
+ None => {
+ warn!(
+ "Row in table '{table}' has no primary-key or tracking value; \
+ using a random message id (replays will not be idempotent)"
+ );
+ Uuid::new_v4().as_u128()
+ }
+ }
+}
+
+fn to_snake_case(input: &str) -> String {
+ let mut result = String::new();
+ let mut prev_was_uppercase = false;
+ for (i, ch) in input.chars().enumerate() {
+ if ch.is_uppercase() {
+ if i > 0 && !prev_was_uppercase {
+ result.push('_');
+ }
+ if let Some(lc) = ch.to_lowercase().next() {
+ result.push(lc);
+ } else {
+ result.push(ch);
+ }
+ prev_was_uppercase = true;
+ } else {
+ result.push(ch);
+ prev_was_uppercase = false;
+ }
+ }
+ result
+}
+
+fn redact_connection_string(conn_str: &str) -> String {
+ if let Some(scheme_end) = conn_str.find("://") {
+ let scheme = &conn_str[..scheme_end + 3];
+ let rest = &conn_str[scheme_end + 3..];
+ let preview: String = rest.chars().take(3).collect();
+ return format!("{scheme}{preview}***");
+ }
+ let preview: String = conn_str.chars().take(3).collect();
+ format!("{preview}***")
+}
+
+fn quote_identifier(name: &str) -> Result<String, Error> {
Review Comment:
Both residual issues are fixed. Substitution is now single-pass like postgres_source's substitute_query_tokens, so substituted values are never rescanned.
NO_BACKSLASH_ESCAPES: the pool's after_connect now strips that mode from every session, so the backslash and NUL escaping in format_offset_value always means what it assumes, regardless of the server's global sql_mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r4187441046
##########
core/connectors/sources/mysql_source/Cargo.toml:
##########
@@ -0,0 +1,57 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[package]
+name = "iggy_connector_mysql_source"
+version = "0.1.0"
+description = "Iggy MySQL source connector supporting table polling for message streaming platform"
+edition = "2024"
+license = "Apache-2.0"
+keywords = ["iggy", "messaging", "streaming", "mysql", "polling"]
+categories = ["command-line-utilities", "database", "network-programming"]
+homepage = "https://iggy.apache.org"
+documentation = "https://iggy.apache.org/docs"
+repository = "https://github.com/apache/iggy"
+readme = "README.md"
+publish = false
+
+[package.metadata.cargo-machete]
+ignored = ["dashmap"]
+
+[lib]
+crate-type = ["cdylib", "lib"]
+
+[dependencies]
+async-trait = { workspace = true }
+base64 = { workspace = true }
+chrono = { workspace = true }
+dashmap = { workspace = true }
+humantime = { workspace = true }
+iggy_common = { workspace = true }
+iggy_connector_sdk = { workspace = true }
+rust_decimal = "=1.42.1"
Review Comment:
fixed. rust_decimal is now declared in [workspace.dependencies] and the crate uses { workspace = true }, so mysql_source has no non-workspace dependencies left.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r4256812571
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,4071 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source,
+ source::SourceBatchResult, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+/// Upper bound on applying a batch's staged mark/delete after its Ack. The SDK waits
+/// `BATCH_RESULT_TIMEOUT` (30s) for the result, so this has to finish well inside it.
+const ACK_BATCH_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// MySQL rejects a prepared statement with more placeholders than this (error 1390), and
+/// the mark/delete binds one per row of the batch.
+const MAX_CLEANUP_BATCH_SIZE: u32 = 65_535;
+
+/// How many mark/delete statements in a row may fail before the connector stops. The
+/// operations stay in the persisted state and run again on restart.
+const MAX_CONSECUTIVE_CLEANUP_FAILURES: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ pending_batch: Mutex<Option<PendingBatch>>,
+ consecutive_cleanup_failures: AtomicU32,
+ verified_cleanup_keys: Mutex<HashSet<String>>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some("it has too few distinct values to be unique per row");
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some("binary values are compared as base64 text, which does not keep byte order");
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some("MySQL orders these by declaration index, not by label");
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL \
+ or not an ordered scalar (tinyint(1), JSON)"
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "row '{value}' is below the stored offset '{previous}' for tracking_column \
+ '{column}'. Filter the query with `{column} > $offset`"
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. Order the query ascending by '{column}'"
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+ /// Mark/delete work for a batch the runtime has already sent. It is persisted with
+ /// the batch's offsets so a crash between the send and the cleanup reruns it.
+ #[serde(default)]
+ pending_operations: Vec<PendingOperation>,
+}
+
+/// The rows of one table to mark or delete once their batch is acknowledged.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct PendingOperation {
+ table: String,
+ ids: Vec<PkValue>,
+ target: CleanupTarget,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct CleanupTarget {
+ key_column: String,
+ action: CleanupAction,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum CleanupAction {
+ Delete,
+ MarkProcessed { column: String },
+}
+
+/// The candidate state of the batch most recently handed to the runtime. Live state
+/// only takes it over on Ack, so a Nack leaves offsets and rows where they were.
+#[derive(Debug, Clone)]
+struct PendingBatch {
+ state: State,
+ /// The runtime has delivered the batch but its cleanup has not finished yet.
+ acknowledged: bool,
+ /// Cleanup finished and the next poll must persist `state` without the operations,
+ /// so a restart does not run them again.
+ retiring: bool,
+}
+
+struct PolledBatch {
+ messages: Vec<ProducedMessage>,
+ pending: Option<PendingBatch>,
+}
+
+enum CleanupKeyCheck {
+ Unique,
+ NotUnique,
+ TableMissing,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty, list the table even when custom_query hardcodes it"
+ .to_string(),
+ ));
+ }
+
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ if let Some(batch_size) = self.config.batch_size
+ && batch_size > MAX_CLEANUP_BATCH_SIZE
+ && self.cleanup_target().is_some()
+ {
+ return Err(Error::InitError(format!(
+ "batch_size {batch_size} exceeds {MAX_CLEANUP_BATCH_SIZE} with delete_after_read or \
+ processed_column set, MySQL allows at most {MAX_CLEANUP_BATCH_SIZE} placeholders \
+ in the statement that marks or deletes a batch"
+ )));
+ }
+
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "name the column custom_query orders by and filters with $offset"
+ } else {
+ "use a unique, monotonically increasing column"
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set, {guidance}"
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.validate_pending_cleanup()?;
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+ self.validate_cleanup_keys().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ if let Some(state) = self.retiring_state().await {
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: Some(self.serialize_pending_state(&state)?),
+ });
+ }
+
+ // Skip the pacing delay while draining a backlog, but never while a cleanup
+ // retry is pending, or a failing mark/delete would spin without a pause.
+ let cleanup_pending = self.pending_batch.lock().await.is_some();
+ if cleanup_pending || !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ {
+ let pending = self.pending_batch.lock().await;
+ if let Some(pending) = pending.as_ref() {
+ if !pending.acknowledged {
+ error!(
+ "{CONNECTOR_NAME} connector ID: {} was polled while a batch was still in flight",
+ self.id
+ );
+ return Err(Error::InvalidState);
+ }
+ // An empty batch whose Ack retries the cleanup the last one left behind.
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: None,
+ });
+ }
+ }
+
+ let polled = self.poll_tables().await?;
+
+ let processed_rows = match &polled.pending {
+ Some(pending) => pending.state.processed_rows,
+ None => self.state.lock().await.processed_rows,
+ };
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ }
+
+ // Idle cycles stage nothing, so return None to let the runtime skip the fsync.
+ let persisted_state = polled
+ .pending
+ .as_ref()
+ .map(|pending| self.serialize_pending_state(&pending.state))
+ .transpose()?;
+ *self.pending_batch.lock().await = polled.pending;
+
+ Ok(ProducedMessages {
+ schema,
+ messages: polled.messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> {
+ if result == SourceBatchResult::Nack {
+ // Dropping the candidate leaves the live offsets where they were, so the
+ // next poll reads the same rows again. Delivered batches keep their cleanup.
+ let mut pending = self.pending_batch.lock().await;
+ if pending
+ .as_ref()
+ .is_some_and(|pending| !pending.acknowledged && !pending.retiring)
+ {
+ pending.take();
+ }
+ return Ok(());
+ }
+
+ let Some(PendingBatch { mut state, .. }) = self.pending_batch.lock().await.clone() else {
+ return Ok(());
+ };
+
+ let operations = std::mem::take(&mut state.pending_operations);
+ let had_operations = !operations.is_empty();
+ let deadline = tokio::time::Instant::now() + ACK_BATCH_TIMEOUT;
+ let failed_operations = self.apply_pending_operations(operations, deadline).await?;
+
+ if failed_operations.is_empty() {
+ *self.state.lock().await = state.clone();
+ let mut pending = self.pending_batch.lock().await;
+ if had_operations {
+ *pending = Some(PendingBatch {
+ state,
+ acknowledged: false,
+ retiring: true,
+ });
+ } else {
+ pending.take();
+ }
+ } else {
+ state.pending_operations = failed_operations;
+ *self.pending_batch.lock().await = Some(PendingBatch {
+ state,
+ acknowledged: true,
+ retiring: false,
+ });
+ }
+
+ Ok(())
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ let state = restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ pending_operations: Vec::new(),
+ });
+ // The runtime only persists a batch's operations after sending it, so restored
+ // operations belong to a delivered batch and only the cleanup is left to do.
+ let pending_batch = (!state.pending_operations.is_empty()).then(|| PendingBatch {
+ state: state.clone(),
+ acknowledged: true,
+ retiring: false,
+ });
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(state),
+ pending_batch: Mutex::new(pending_batch),
+ consecutive_cleanup_failures: AtomicU32::new(0),
+ verified_cleanup_keys: Mutex::new(HashSet::new()),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ poisoned_tables: Mutex::new(HashSet::new()),
+ tracking_kinds: Mutex::new(HashMap::new()),
+ polls: AtomicU64::new(0),
+ cursor_regressions: Mutex::new(HashMap::new()),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ // `format_offset_value` escapes backslashes, which under NO_BACKSLASH_ESCAPES
+ // would reach MySQL as literal characters and filter on the wrong cursor.
+ .after_connect(|connection, _| {
+ Box::pin(async move {
+ sqlx::query(
+ "SET SESSION sql_mode = REPLACE(@@SESSION.sql_mode, 'NO_BACKSLASH_ESCAPES', '')",
+ )
+ .execute(&mut *connection)
+ .await?;
+ Ok(())
+ })
+ })
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn tracking_column(&self) -> &str {
+ self.config.tracking_column.as_deref().unwrap_or("id")
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn resolve_tracking_columns(&self) -> Result<HashMap<String, OffsetKind>, Error> {
+ // Whether the connector owns the query, and can therefore hold the schema to
+ // what that query will do with it.
+ let strict = self.config.custom_query.is_none();
+ let pool = self.get_pool()?;
+ let tracking_column = self.tracking_column();
+ let mut kinds = HashMap::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
Review Comment:
critical: `BINARY` tracking columns in tables created after startup bypass validation, so base64 cursors skip rows when compared with raw bytes. reject unsupported tracking types from result metadata before advancing the cursor.
##########
core/connectors/sources/mysql_source/README.md:
##########
@@ -0,0 +1,631 @@
+# MySQL Source Connector
+
+The MySQL source connector fetches data from MySQL databases and streams it to Iggy topics. It supports incremental table polling with flexible payload extraction.
+
+> **Note:** Only polling mode is available. Binlog CDC support is planned for a future release.
+
+## Features
+
+- **Table Polling**: Incrementally fetch data from MySQL tables using a tracking column
+- **Flexible Payload Extraction**: Extract BLOB, TEXT, or JSON columns directly as payload
+- **Custom Queries**: Use custom SQL queries with parameter substitution
+- **Delete After Read**: Automatically delete rows after processing
+- **Mark as Processed**: Mark rows as processed using a boolean column
+- **Multiple Tables**: Monitor multiple tables, polled one after another each cycle
+- **Batch Processing**: Fetch data in configurable batch sizes
+- **Offset Tracking**: Resume incremental polling from the last processed tracking value (see [Delivery semantics](#delivery-semantics))
+
+## Configuration
+
+```toml
+[plugin_config]
+connection_string = "mysql://user:pass@localhost:3306/database"
+tables = ["users", "orders"]
+poll_interval = "1s"
+batch_size = 1000
+tracking_column = "id"
+initial_offset = "0"
+max_connections = 10
+snake_case_columns = false
+include_metadata = true
+
+# Payload extraction (optional)
+payload_column = "payload"
+payload_format = "bytea"
+
+# Delete/mark processed (optional)
+delete_after_read = false
+processed_column = "is_processed"
+primary_key_column = "id"
+
+# Custom query (optional)
+custom_query = "SELECT * FROM $table WHERE id > $offset ORDER BY id LIMIT $limit"
+```
+
+## Configuration Options
+
+| Option | Type | Default | Description |
+| ------ | ---- | ------- | ----------- |
+| `connection_string` | string | required | MySQL connection string (`mysql://user:pass@host:3306/db`) |
+| `tables` | array | required | List of tables to monitor |
+| `poll_interval` | string | `10s` | How often to poll (e.g., `1s`, `5m`) |
+| `batch_size` | u32 | `1000` | Max rows per poll; must be greater than 0, greater than 1 when `custom_query` filters on `$offset`, and at most 65535 when `delete_after_read` or `processed_column` is set (see [Ordering is checked on the rows](#ordering-is-checked-on-the-rows)) |
+| `tracking_column` | string | required | Column for incremental polling; must be unique and monotonically increasing (see [Tracking Column Requirements](#tracking-column-requirements)) |
+| `initial_offset` | string | none | Starting value for tracking column |
+| `max_connections` | u32 | `10` | Max database connections |
+| `snake_case_columns` | bool | `false` | Convert column names to snake_case |
+| `include_metadata` | bool | `true` | Wrap results with metadata envelope |
+| `payload_column` | string | none | Column to extract directly as payload |
+| `payload_format` | string | `json` (invalid if `payload_column` is set — `payload_format` is required in that case) | Format of payload_column: `bytea`, `text`, or `json_direct` |
+| `delete_after_read` | bool | `false` | Delete rows after reading |
+| `processed_column` | string | none | Boolean column to mark as processed |
+| `primary_key_column` | string | tracking_column | Key for delete/mark operations; must back a single-column `PRIMARY KEY` or `UNIQUE` index (see [Delete After Read / Mark as Processed](#delete-after-read--mark-as-processed)) |
+| `custom_query` | string | none | Custom SQL with parameter substitution; must be a read, and must either filter on `$offset` with an `ORDER BY` or consume its rows (see [What the query has to be](#what-the-query-has-to-be)) |
+| `verbose_logging` | bool | `false` | Log at info level instead of debug |
+| `max_retries` | u32 | `3` | Retries after the initial attempt for transient errors (3 = 4 total attempts) |
+| `retry_delay` | string | `1s` | Base delay between retries (e.g., `500ms`, `2s`) |
+
+## Tracking Column Requirements
+
+Polling is **insert-only**. Each poll runs roughly `SELECT ... WHERE tracking_column > last_offset ORDER BY tracking_column ASC LIMIT batch_size` and stores the largest tracking value it saw as the next offset. For this to be lossless the **`tracking_column` must be unique and monotonically increasing under the order MySQL sorts that column in** — an auto-increment primary key is the canonical choice.
Review Comment:
warning: an `AUTO_INCREMENT` id can commit after a higher id advances the cursor, so the lossless polling claim is too broad. document commit-order visibility requirements and limit the delivery guarantee to selected rows.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,4071 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source,
+ source::SourceBatchResult, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+/// Upper bound on applying a batch's staged mark/delete after its Ack. The SDK waits
+/// `BATCH_RESULT_TIMEOUT` (30s) for the result, so this has to finish well inside it.
+const ACK_BATCH_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// MySQL rejects a prepared statement with more placeholders than this (error 1390), and
+/// the mark/delete binds one per row of the batch.
+const MAX_CLEANUP_BATCH_SIZE: u32 = 65_535;
+
+/// How many mark/delete statements in a row may fail before the connector stops. The
+/// operations stay in the persisted state and run again on restart.
+const MAX_CONSECUTIVE_CLEANUP_FAILURES: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ pending_batch: Mutex<Option<PendingBatch>>,
+ consecutive_cleanup_failures: AtomicU32,
+ verified_cleanup_keys: Mutex<HashSet<String>>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some("it has too few distinct values to be unique per row");
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some("binary values are compared as base64 text, which does not keep byte order");
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some("MySQL orders these by declaration index, not by label");
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL \
+ or not an ordered scalar (tinyint(1), JSON)"
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "row '{value}' is below the stored offset '{previous}' for tracking_column \
+ '{column}'. Filter the query with `{column} > $offset`"
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. Order the query ascending by '{column}'"
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+ /// Mark/delete work for a batch the runtime has already sent. It is persisted with
+ /// the batch's offsets so a crash between the send and the cleanup reruns it.
+ #[serde(default)]
+ pending_operations: Vec<PendingOperation>,
+}
+
+/// The rows of one table to mark or delete once their batch is acknowledged.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct PendingOperation {
+ table: String,
+ ids: Vec<PkValue>,
+ target: CleanupTarget,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct CleanupTarget {
+ key_column: String,
+ action: CleanupAction,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum CleanupAction {
+ Delete,
+ MarkProcessed { column: String },
+}
+
+/// The candidate state of the batch most recently handed to the runtime. Live state
+/// only takes it over on Ack, so a Nack leaves offsets and rows where they were.
+#[derive(Debug, Clone)]
+struct PendingBatch {
+ state: State,
+ /// The runtime has delivered the batch but its cleanup has not finished yet.
+ acknowledged: bool,
+ /// Cleanup finished and the next poll must persist `state` without the operations,
+ /// so a restart does not run them again.
+ retiring: bool,
+}
+
+struct PolledBatch {
+ messages: Vec<ProducedMessage>,
+ pending: Option<PendingBatch>,
+}
+
+enum CleanupKeyCheck {
+ Unique,
+ NotUnique,
+ TableMissing,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty, list the table even when custom_query hardcodes it"
+ .to_string(),
+ ));
+ }
+
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ if let Some(batch_size) = self.config.batch_size
+ && batch_size > MAX_CLEANUP_BATCH_SIZE
+ && self.cleanup_target().is_some()
+ {
+ return Err(Error::InitError(format!(
+ "batch_size {batch_size} exceeds {MAX_CLEANUP_BATCH_SIZE} with delete_after_read or \
+ processed_column set, MySQL allows at most {MAX_CLEANUP_BATCH_SIZE} placeholders \
+ in the statement that marks or deletes a batch"
+ )));
+ }
+
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "name the column custom_query orders by and filters with $offset"
+ } else {
+ "use a unique, monotonically increasing column"
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set, {guidance}"
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.validate_pending_cleanup()?;
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+ self.validate_cleanup_keys().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ if let Some(state) = self.retiring_state().await {
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: Some(self.serialize_pending_state(&state)?),
+ });
+ }
+
+ // Skip the pacing delay while draining a backlog, but never while a cleanup
+ // retry is pending, or a failing mark/delete would spin without a pause.
+ let cleanup_pending = self.pending_batch.lock().await.is_some();
+ if cleanup_pending || !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ {
+ let pending = self.pending_batch.lock().await;
+ if let Some(pending) = pending.as_ref() {
+ if !pending.acknowledged {
+ error!(
+ "{CONNECTOR_NAME} connector ID: {} was polled while a batch was still in flight",
+ self.id
+ );
+ return Err(Error::InvalidState);
+ }
+ // An empty batch whose Ack retries the cleanup the last one left behind.
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: None,
+ });
+ }
+ }
+
+ let polled = self.poll_tables().await?;
+
+ let processed_rows = match &polled.pending {
+ Some(pending) => pending.state.processed_rows,
+ None => self.state.lock().await.processed_rows,
+ };
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ }
+
+ // Idle cycles stage nothing, so return None to let the runtime skip the fsync.
+ let persisted_state = polled
+ .pending
+ .as_ref()
+ .map(|pending| self.serialize_pending_state(&pending.state))
+ .transpose()?;
+ *self.pending_batch.lock().await = polled.pending;
+
+ Ok(ProducedMessages {
+ schema,
+ messages: polled.messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> {
+ if result == SourceBatchResult::Nack {
+ // Dropping the candidate leaves the live offsets where they were, so the
+ // next poll reads the same rows again. Delivered batches keep their cleanup.
+ let mut pending = self.pending_batch.lock().await;
+ if pending
+ .as_ref()
+ .is_some_and(|pending| !pending.acknowledged && !pending.retiring)
+ {
+ pending.take();
+ }
+ return Ok(());
+ }
+
+ let Some(PendingBatch { mut state, .. }) = self.pending_batch.lock().await.clone() else {
+ return Ok(());
+ };
+
+ let operations = std::mem::take(&mut state.pending_operations);
+ let had_operations = !operations.is_empty();
+ let deadline = tokio::time::Instant::now() + ACK_BATCH_TIMEOUT;
+ let failed_operations = self.apply_pending_operations(operations, deadline).await?;
+
+ if failed_operations.is_empty() {
+ *self.state.lock().await = state.clone();
+ let mut pending = self.pending_batch.lock().await;
+ if had_operations {
+ *pending = Some(PendingBatch {
+ state,
+ acknowledged: false,
+ retiring: true,
+ });
+ } else {
+ pending.take();
+ }
+ } else {
+ state.pending_operations = failed_operations;
+ *self.pending_batch.lock().await = Some(PendingBatch {
+ state,
+ acknowledged: true,
+ retiring: false,
+ });
+ }
+
+ Ok(())
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ let state = restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ pending_operations: Vec::new(),
+ });
+ // The runtime only persists a batch's operations after sending it, so restored
+ // operations belong to a delivered batch and only the cleanup is left to do.
+ let pending_batch = (!state.pending_operations.is_empty()).then(|| PendingBatch {
+ state: state.clone(),
+ acknowledged: true,
+ retiring: false,
+ });
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(state),
+ pending_batch: Mutex::new(pending_batch),
+ consecutive_cleanup_failures: AtomicU32::new(0),
+ verified_cleanup_keys: Mutex::new(HashSet::new()),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ poisoned_tables: Mutex::new(HashSet::new()),
+ tracking_kinds: Mutex::new(HashMap::new()),
+ polls: AtomicU64::new(0),
+ cursor_regressions: Mutex::new(HashMap::new()),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ // `format_offset_value` escapes backslashes, which under NO_BACKSLASH_ESCAPES
+ // would reach MySQL as literal characters and filter on the wrong cursor.
+ .after_connect(|connection, _| {
+ Box::pin(async move {
+ sqlx::query(
+ "SET SESSION sql_mode = REPLACE(@@SESSION.sql_mode, 'NO_BACKSLASH_ESCAPES', '')",
+ )
+ .execute(&mut *connection)
+ .await?;
+ Ok(())
+ })
+ })
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn tracking_column(&self) -> &str {
+ self.config.tracking_column.as_deref().unwrap_or("id")
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn resolve_tracking_columns(&self) -> Result<HashMap<String, OffsetKind>, Error> {
+ // Whether the connector owns the query, and can therefore hold the schema to
+ // what that query will do with it.
+ let strict = self.config.custom_query.is_none();
+ let pool = self.get_pool()?;
+ let tracking_column = self.tracking_column();
+ let mut kinds = HashMap::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ }
+
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ }
+
+ warn!(
+ "tracking_column '{tracking_column}' is not a column of table '{table}'; \
+ assuming custom_query projects it. Its comparison order is resolved from \
+ the first result set instead."
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ // The column's type survives a plain projection, so it is worth rejecting
+ // whatever the query looks like. A custom_query aliasing an expression onto
+ // a name the table also uses is the one false positive, hence the hint.
+ if let Some(reason) = unusable_tracking_type(&data_type) {
+ let aliasing_hint = if self.config.custom_query.is_some() {
+ " If custom_query aliases a different expression to this name, rename the \
+ alias so it does not collide with the column."
+ } else {
+ ""
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': {reason}. Use an integer, timestamp, or string \
+ column.{aliasing_hint}"
+ )));
+ }
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; a row \
+ whose value is NULL cannot produce a cursor and will fail that table's batch. \
+ Have custom_query exclude those rows, or declare the column NOT NULL."
+ );
+ }
+
+ if is_collation_ordered(&data_type) {
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': values must be unique and increase in its collation order \
+ (zero-pad numbers, prefer a _bin collation)"
+ );
+ }
+
+ kinds.insert(table.clone(), offset_kind_for_data_type(&data_type));
+ }
+ Ok(kinds)
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn serialize_pending_state(&self, state: &State) -> Result<ConnectorState, Error> {
+ self.serialize_state(state).ok_or_else(|| {
+ Error::Serialization(format!("failed to serialize {CONNECTOR_NAME} state"))
+ })
+ }
+
+ async fn retiring_state(&self) -> Option<State> {
+ self.pending_batch
+ .lock()
+ .await
+ .as_ref()
+ .filter(|pending| pending.retiring)
+ .map(|pending| pending.state.clone())
+ }
+
+ fn primary_key_column(&self) -> &str {
+ self.config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(self.tracking_column())
+ }
+
+ fn cleanup_target(&self) -> Option<CleanupTarget> {
+ let action = if self.config.delete_after_read.unwrap_or(false) {
+ CleanupAction::Delete
+ } else if let Some(column) = &self.config.processed_column {
+ CleanupAction::MarkProcessed {
+ column: column.clone(),
+ }
+ } else {
+ return None;
+ };
+ Some(CleanupTarget {
+ key_column: self.primary_key_column().to_string(),
+ action,
+ })
+ }
+
+ /// Restored operations were staged under the config of the run that wrote them.
+ /// Running them under a different key or action would touch rows nobody read.
+ fn validate_pending_cleanup(&mut self) -> Result<(), Error> {
+ let configured = self.cleanup_target();
+ for operation in &self.state.get_mut().pending_operations {
+ if configured.as_ref() != Some(&operation.target) {
+ return Err(Error::InitError(format!(
+ "the cleanup configuration changed while cleanup of {} delivered row(s) in \
+ table '{}' is pending; restore primary_key_column '{}' and action {:?}",
+ operation.ids.len(),
+ operation.table,
+ operation.target.key_column,
+ operation.target.action
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ /// `WHERE key IN (...)` on a key that repeats marks or deletes every row sharing a
+ /// value, including ones that were never read. Tables that don't exist yet are
+ /// checked by the first poll that stages cleanup for them.
+ async fn validate_cleanup_keys(&mut self) -> Result<(), Error> {
+ let Some(target) = self.cleanup_target() else {
+ return Ok(());
+ };
+ let pool = self.get_pool()?;
+ let mut verified = HashSet::with_capacity(self.config.tables.len());
+ for table in &self.config.tables {
+ let check = cleanup_key_check(pool, table, &target.key_column)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to check primary_key_column '{}' on table '{table}': {e}",
+ target.key_column
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ verified.insert(table.clone());
+ }
+ CleanupKeyCheck::NotUnique => {
+ return Err(Error::InitError(not_unique_key_message(
+ table,
+ &target.key_column,
+ )));
+ }
+ CleanupKeyCheck::TableMissing => {}
+ }
+ }
+ *self.verified_cleanup_keys.get_mut() = verified;
+ Ok(())
+ }
+
+ async fn cleanup_key_is_unique(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ key_column: &str,
+ ) -> Result<bool, Error> {
+ if self.verified_cleanup_keys.lock().await.contains(table) {
+ return Ok(true);
+ }
+ let check = cleanup_key_check(pool, table, key_column)
+ .await
+ .map_err(|e| {
+ Error::Connection(format!(
+ "failed to check primary_key_column '{key_column}' on table '{table}': {e}"
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ self.verified_cleanup_keys
+ .lock()
+ .await
+ .insert(table.to_string());
+ Ok(true)
+ }
+ CleanupKeyCheck::NotUnique => Ok(false),
+ CleanupKeyCheck::TableMissing => Err(Error::Connection(format!(
+ "table '{table}' disappeared before its primary_key_column could be checked"
+ ))),
+ }
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ /// Best current knowledge of how a table's tracking values compare.
+ async fn tracking_kind(&self, table: &str) -> OffsetKind {
+ self.tracking_kinds
+ .lock()
+ .await
+ .get(table)
+ .copied()
+ .unwrap_or(OffsetKind::Unknown)
+ }
+
+ /// Whether the stored cursor is what limits the rows a poll can see. The built
+ /// polling query always filters on it; a `custom_query` does so only if it
+ /// interpolates `$offset`.
+ fn cursor_gates_query(&self) -> bool {
+ match &self.config.custom_query {
+ Some(query) => query.contains("$offset"),
+ None => true,
+ }
+ }
+
+ /// The offset a poll actually resumes from: the stored cursor, or the configured
+ /// starting point before anything has been stored.
+ fn effective_offset(&self, last_offset: &Option<String>) -> Option<String> {
+ last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let offset_value = self.effective_offset(last_offset).unwrap_or_default();
+ let offset = format_offset_value(&offset_value, kind);
+ let quoted_table = quote_qualified_identifier(table)?;
+ let limit = batch_size.to_string();
+ let now = Utc::now();
+ let now_unix = now.timestamp().to_string();
+ let now = now.to_rfc3339();
+ let replacements = [
+ ("$table", quoted_table.as_str()),
+ ("$offset", offset.as_str()),
+ ("$limit", limit.as_str()),
+ ("$now_unix", now_unix.as_str()),
+ ("$now", now.as_str()),
+ ];
+
+ Ok(substitute_query_tokens(query, &replacements))
+ }
+
+ /// The query is substituted as text and re-executed once per configured table on
+ /// every poll for the life of the process, so a shape that cannot terminate does
+ /// not fail, it repeats. What can be judged from the text alone is judged here,
+ /// at startup, rather than left to the first poll.
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let tracking_column = self.tracking_column();
+
+ match leading_statement_token(query) {
+ None => {
+ return Err(Error::InitError(
+ "custom_query has no statement: it is empty, or nothing but comments"
+ .to_string(),
+ ));
+ }
+ Some(token) if !READ_STATEMENT_STARTS.contains(&token.as_str()) => {
+ return Err(Error::InitError(format!(
+ "custom_query must be a read starting with SELECT, WITH or TABLE, found \
+ '{token}'"
+ )));
+ }
+ Some(_) => {}
+ }
+
+ // Collapsed so `ORDER\nBY` and `ORDER BY` read the same as `ORDER BY`; the
+ // check hard-fails, so a formatting difference must not decide it.
+ let normalized = query
+ .split_whitespace()
+ .collect::<Vec<_>>()
+ .join(" ")
+ .to_uppercase();
+
+ if query.contains("$offset") {
+ if !normalized.contains("ORDER BY") {
+ return Err(Error::InitError(format!(
+ "custom_query filters on $offset but has no ORDER BY, add \
+ `ORDER BY {tracking_column}`"
+ )));
+ }
+ } else if !self.config.delete_after_read.unwrap_or(false)
+ && self.config.processed_column.is_none()
+ {
+ return Err(Error::InitError(format!(
+ "custom_query must filter with $offset (`WHERE {tracking_column} > $offset`), or \
+ set delete_after_read or processed_column"
+ )));
+ }
+
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "custom_query uses $table but no tables are configured".to_string(),
+ ));
+ }
+ // Without $table the same SQL is executed once per table, so identical rows
+ // would be emitted repeatedly, each carrying a different table's metadata,
+ // message ID, and tracking offset. Table identity is not recoverable from a
+ // static query, so reject the combination instead of guessing one.
+ if !query.contains("$table") && self.config.tables.len() > 1 {
+ return Err(Error::InitError(format!(
+ "custom_query has no $table placeholder but {} tables are configured ({}), use \
+ $table in the query or configure a single table",
+ self.config.tables.len(),
+ self.config.tables.join(", ")
+ )));
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = self.effective_offset(last_offset) {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(&offset, kind)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn apply_pending_operations(
+ &self,
+ operations: Vec<PendingOperation>,
+ deadline: tokio::time::Instant,
+ ) -> Result<Vec<PendingOperation>, Error> {
+ let mut failed_operations = Vec::new();
+ let mut operations = operations.into_iter();
+ while let Some(operation) = operations.next() {
+ if tokio::time::Instant::now() >= deadline {
+ failed_operations.push(operation);
+ failed_operations.extend(operations);
+ break;
+ }
+ let result = tokio::time::timeout_at(deadline, async {
+ let pool = self.get_pool()?;
+ self.mark_or_delete_processed_rows(pool, &operation).await
+ })
+ .await;
+ match result {
+ Ok(Ok(())) => {
+ self.consecutive_cleanup_failures
+ .store(0, Ordering::Relaxed);
+ }
+ Ok(Err(error)) => {
+ self.record_cleanup_failure(&operation, &error)?;
+ failed_operations.push(operation);
+ }
+ Err(_) => {
+ self.record_cleanup_failure(
+ &operation,
+ "cleanup exceeded the 10s budget for one Ack",
+ )?;
+ failed_operations.push(operation);
+ }
+ }
+ }
+ Ok(failed_operations)
+ }
+
+ fn record_cleanup_failure(
+ &self,
+ operation: &PendingOperation,
+ error: impl std::fmt::Display,
+ ) -> Result<(), Error> {
+ let consecutive_failures = self
+ .consecutive_cleanup_failures
+ .fetch_add(1, Ordering::Relaxed)
+ .saturating_add(1);
+ error!(
+ "Failed to mark or delete {} delivered row(s) in table '{}' for {CONNECTOR_NAME} \
+ connector ID: {}, retrying on the next poll ({consecutive_failures} consecutive \
+ failures): {error}",
+ operation.ids.len(),
+ operation.table,
+ self.id
+ );
+ if consecutive_failures >= MAX_CONSECUTIVE_CLEANUP_FAILURES {
+ return Err(Error::Connection(format!(
+ "stopping {CONNECTOR_NAME} connector ID {} after {consecutive_failures} \
+ consecutive mark/delete failures; the cleanup stays in the saved state and \
+ runs again on restart",
+ self.id
+ )));
+ }
+ Ok(())
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ operation: &PendingOperation,
+ ) -> Result<(), Error> {
+ let PendingOperation { table, ids, target } = operation;
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(&target.key_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = match &target.action {
+ CleanupAction::Delete => {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
+ }
+ CleanupAction::MarkProcessed { column } => {
+ let quoted_processed = quote_identifier(column)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({placeholders})"
+ )
+ }
+ };
+
+ // The query and its binds are rebuilt on every attempt because with_retry
+ // calls the closure once per try and each `execute` consumes the query.
+ with_retry(
+ || {
+ let mut prepared = sqlx::query(sqlx::AssertSqlSafe(query.as_str()));
+ for id in ids {
+ prepared = match id {
+ PkValue::Text(text) => prepared.bind(text),
+ PkValue::Bytes(bytes) => prepared.bind(bytes),
+ };
+ }
+ prepared.execute(pool)
+ },
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<PolledBatch, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.tracking_column();
+ let pk_column = self.primary_key_column();
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ //
+ // An ordering violation is handled one step further. Retrying it would
+ // re-run the same query, re-decode the same rows, and hit the same failure
+ // forever, so the table is latched off instead of being retried each cycle.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ let poll_number = self.polls.fetch_add(1, Ordering::Relaxed) + 1;
+ let poisoned = { self.poisoned_tables.lock().await.clone() };
+ let mut newly_poisoned: Vec<String> = Vec::new();
+
+ for table in &self.config.tables {
+ if poisoned.contains(table) {
+ continue;
+ }
+
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ if !batch.messages.is_empty() {
+ self.cursor_regressions.lock().await.remove(table);
+ }
+ // A full batch means the LIMIT was hit, so more rows are waiting.
+ any_batch_full |= batch.messages.len() as u32 >= batch_size;
Review Comment:
warning: a discarded full batch still sets `last_batch_full`, so a missing projected cleanup key causes repeated fetches without a pause. set it only for batches accepted into pending state.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,4071 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source,
+ source::SourceBatchResult, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+/// Upper bound on applying a batch's staged mark/delete after its Ack. The SDK waits
+/// `BATCH_RESULT_TIMEOUT` (30s) for the result, so this has to finish well inside it.
+const ACK_BATCH_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// MySQL rejects a prepared statement with more placeholders than this (error 1390), and
+/// the mark/delete binds one per row of the batch.
+const MAX_CLEANUP_BATCH_SIZE: u32 = 65_535;
+
+/// How many mark/delete statements in a row may fail before the connector stops. The
+/// operations stay in the persisted state and run again on restart.
+const MAX_CONSECUTIVE_CLEANUP_FAILURES: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ pending_batch: Mutex<Option<PendingBatch>>,
+ consecutive_cleanup_failures: AtomicU32,
+ verified_cleanup_keys: Mutex<HashSet<String>>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some("it has too few distinct values to be unique per row");
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some("binary values are compared as base64 text, which does not keep byte order");
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some("MySQL orders these by declaration index, not by label");
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL \
+ or not an ordered scalar (tinyint(1), JSON)"
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "row '{value}' is below the stored offset '{previous}' for tracking_column \
+ '{column}'. Filter the query with `{column} > $offset`"
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. Order the query ascending by '{column}'"
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+ /// Mark/delete work for a batch the runtime has already sent. It is persisted with
+ /// the batch's offsets so a crash between the send and the cleanup reruns it.
+ #[serde(default)]
+ pending_operations: Vec<PendingOperation>,
+}
+
+/// The rows of one table to mark or delete once their batch is acknowledged.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct PendingOperation {
+ table: String,
+ ids: Vec<PkValue>,
+ target: CleanupTarget,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct CleanupTarget {
+ key_column: String,
+ action: CleanupAction,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum CleanupAction {
+ Delete,
+ MarkProcessed { column: String },
+}
+
+/// The candidate state of the batch most recently handed to the runtime. Live state
+/// only takes it over on Ack, so a Nack leaves offsets and rows where they were.
+#[derive(Debug, Clone)]
+struct PendingBatch {
+ state: State,
+ /// The runtime has delivered the batch but its cleanup has not finished yet.
+ acknowledged: bool,
+ /// Cleanup finished and the next poll must persist `state` without the operations,
+ /// so a restart does not run them again.
+ retiring: bool,
+}
+
+struct PolledBatch {
+ messages: Vec<ProducedMessage>,
+ pending: Option<PendingBatch>,
+}
+
+enum CleanupKeyCheck {
+ Unique,
+ NotUnique,
+ TableMissing,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty, list the table even when custom_query hardcodes it"
+ .to_string(),
+ ));
+ }
+
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ if let Some(batch_size) = self.config.batch_size
+ && batch_size > MAX_CLEANUP_BATCH_SIZE
+ && self.cleanup_target().is_some()
+ {
+ return Err(Error::InitError(format!(
+ "batch_size {batch_size} exceeds {MAX_CLEANUP_BATCH_SIZE} with delete_after_read or \
+ processed_column set, MySQL allows at most {MAX_CLEANUP_BATCH_SIZE} placeholders \
+ in the statement that marks or deletes a batch"
+ )));
+ }
+
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "name the column custom_query orders by and filters with $offset"
+ } else {
+ "use a unique, monotonically increasing column"
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set, {guidance}"
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.validate_pending_cleanup()?;
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+ self.validate_cleanup_keys().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ if let Some(state) = self.retiring_state().await {
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: Some(self.serialize_pending_state(&state)?),
+ });
+ }
+
+ // Skip the pacing delay while draining a backlog, but never while a cleanup
+ // retry is pending, or a failing mark/delete would spin without a pause.
+ let cleanup_pending = self.pending_batch.lock().await.is_some();
+ if cleanup_pending || !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ {
+ let pending = self.pending_batch.lock().await;
+ if let Some(pending) = pending.as_ref() {
+ if !pending.acknowledged {
+ error!(
+ "{CONNECTOR_NAME} connector ID: {} was polled while a batch was still in flight",
+ self.id
+ );
+ return Err(Error::InvalidState);
+ }
+ // An empty batch whose Ack retries the cleanup the last one left behind.
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: None,
+ });
+ }
+ }
+
+ let polled = self.poll_tables().await?;
+
+ let processed_rows = match &polled.pending {
+ Some(pending) => pending.state.processed_rows,
+ None => self.state.lock().await.processed_rows,
+ };
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ }
+
+ // Idle cycles stage nothing, so return None to let the runtime skip the fsync.
+ let persisted_state = polled
+ .pending
+ .as_ref()
+ .map(|pending| self.serialize_pending_state(&pending.state))
+ .transpose()?;
+ *self.pending_batch.lock().await = polled.pending;
+
+ Ok(ProducedMessages {
+ schema,
+ messages: polled.messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> {
+ if result == SourceBatchResult::Nack {
+ // Dropping the candidate leaves the live offsets where they were, so the
+ // next poll reads the same rows again. Delivered batches keep their cleanup.
+ let mut pending = self.pending_batch.lock().await;
+ if pending
+ .as_ref()
+ .is_some_and(|pending| !pending.acknowledged && !pending.retiring)
+ {
+ pending.take();
+ }
+ return Ok(());
+ }
+
+ let Some(PendingBatch { mut state, .. }) = self.pending_batch.lock().await.clone() else {
+ return Ok(());
+ };
+
+ let operations = std::mem::take(&mut state.pending_operations);
+ let had_operations = !operations.is_empty();
+ let deadline = tokio::time::Instant::now() + ACK_BATCH_TIMEOUT;
+ let failed_operations = self.apply_pending_operations(operations, deadline).await?;
+
+ if failed_operations.is_empty() {
+ *self.state.lock().await = state.clone();
+ let mut pending = self.pending_batch.lock().await;
+ if had_operations {
+ *pending = Some(PendingBatch {
+ state,
+ acknowledged: false,
+ retiring: true,
+ });
+ } else {
+ pending.take();
+ }
+ } else {
+ state.pending_operations = failed_operations;
+ *self.pending_batch.lock().await = Some(PendingBatch {
+ state,
+ acknowledged: true,
+ retiring: false,
+ });
+ }
+
+ Ok(())
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ let state = restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ pending_operations: Vec::new(),
+ });
+ // The runtime only persists a batch's operations after sending it, so restored
+ // operations belong to a delivered batch and only the cleanup is left to do.
+ let pending_batch = (!state.pending_operations.is_empty()).then(|| PendingBatch {
+ state: state.clone(),
+ acknowledged: true,
+ retiring: false,
+ });
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(state),
+ pending_batch: Mutex::new(pending_batch),
+ consecutive_cleanup_failures: AtomicU32::new(0),
+ verified_cleanup_keys: Mutex::new(HashSet::new()),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ poisoned_tables: Mutex::new(HashSet::new()),
+ tracking_kinds: Mutex::new(HashMap::new()),
+ polls: AtomicU64::new(0),
+ cursor_regressions: Mutex::new(HashMap::new()),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ // `format_offset_value` escapes backslashes, which under NO_BACKSLASH_ESCAPES
+ // would reach MySQL as literal characters and filter on the wrong cursor.
+ .after_connect(|connection, _| {
+ Box::pin(async move {
+ sqlx::query(
+ "SET SESSION sql_mode = REPLACE(@@SESSION.sql_mode, 'NO_BACKSLASH_ESCAPES', '')",
+ )
+ .execute(&mut *connection)
+ .await?;
+ Ok(())
+ })
+ })
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn tracking_column(&self) -> &str {
+ self.config.tracking_column.as_deref().unwrap_or("id")
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn resolve_tracking_columns(&self) -> Result<HashMap<String, OffsetKind>, Error> {
+ // Whether the connector owns the query, and can therefore hold the schema to
+ // what that query will do with it.
+ let strict = self.config.custom_query.is_none();
+ let pool = self.get_pool()?;
+ let tracking_column = self.tracking_column();
+ let mut kinds = HashMap::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ }
+
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ }
+
+ warn!(
+ "tracking_column '{tracking_column}' is not a column of table '{table}'; \
+ assuming custom_query projects it. Its comparison order is resolved from \
+ the first result set instead."
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ // The column's type survives a plain projection, so it is worth rejecting
+ // whatever the query looks like. A custom_query aliasing an expression onto
+ // a name the table also uses is the one false positive, hence the hint.
+ if let Some(reason) = unusable_tracking_type(&data_type) {
+ let aliasing_hint = if self.config.custom_query.is_some() {
+ " If custom_query aliases a different expression to this name, rename the \
+ alias so it does not collide with the column."
+ } else {
+ ""
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': {reason}. Use an integer, timestamp, or string \
+ column.{aliasing_hint}"
+ )));
+ }
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; a row \
+ whose value is NULL cannot produce a cursor and will fail that table's batch. \
+ Have custom_query exclude those rows, or declare the column NOT NULL."
+ );
+ }
+
+ if is_collation_ordered(&data_type) {
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': values must be unique and increase in its collation order \
+ (zero-pad numbers, prefer a _bin collation)"
+ );
+ }
+
+ kinds.insert(table.clone(), offset_kind_for_data_type(&data_type));
+ }
+ Ok(kinds)
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn serialize_pending_state(&self, state: &State) -> Result<ConnectorState, Error> {
+ self.serialize_state(state).ok_or_else(|| {
+ Error::Serialization(format!("failed to serialize {CONNECTOR_NAME} state"))
+ })
+ }
+
+ async fn retiring_state(&self) -> Option<State> {
+ self.pending_batch
+ .lock()
+ .await
+ .as_ref()
+ .filter(|pending| pending.retiring)
+ .map(|pending| pending.state.clone())
+ }
+
+ fn primary_key_column(&self) -> &str {
+ self.config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(self.tracking_column())
+ }
+
+ fn cleanup_target(&self) -> Option<CleanupTarget> {
+ let action = if self.config.delete_after_read.unwrap_or(false) {
+ CleanupAction::Delete
+ } else if let Some(column) = &self.config.processed_column {
+ CleanupAction::MarkProcessed {
+ column: column.clone(),
+ }
+ } else {
+ return None;
+ };
+ Some(CleanupTarget {
+ key_column: self.primary_key_column().to_string(),
+ action,
+ })
+ }
+
+ /// Restored operations were staged under the config of the run that wrote them.
+ /// Running them under a different key or action would touch rows nobody read.
+ fn validate_pending_cleanup(&mut self) -> Result<(), Error> {
+ let configured = self.cleanup_target();
+ for operation in &self.state.get_mut().pending_operations {
+ if configured.as_ref() != Some(&operation.target) {
+ return Err(Error::InitError(format!(
+ "the cleanup configuration changed while cleanup of {} delivered row(s) in \
+ table '{}' is pending; restore primary_key_column '{}' and action {:?}",
+ operation.ids.len(),
+ operation.table,
+ operation.target.key_column,
+ operation.target.action
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ /// `WHERE key IN (...)` on a key that repeats marks or deletes every row sharing a
+ /// value, including ones that were never read. Tables that don't exist yet are
+ /// checked by the first poll that stages cleanup for them.
+ async fn validate_cleanup_keys(&mut self) -> Result<(), Error> {
+ let Some(target) = self.cleanup_target() else {
+ return Ok(());
+ };
+ let pool = self.get_pool()?;
+ let mut verified = HashSet::with_capacity(self.config.tables.len());
+ for table in &self.config.tables {
+ let check = cleanup_key_check(pool, table, &target.key_column)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to check primary_key_column '{}' on table '{table}': {e}",
+ target.key_column
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ verified.insert(table.clone());
+ }
+ CleanupKeyCheck::NotUnique => {
+ return Err(Error::InitError(not_unique_key_message(
+ table,
+ &target.key_column,
+ )));
+ }
+ CleanupKeyCheck::TableMissing => {}
+ }
+ }
+ *self.verified_cleanup_keys.get_mut() = verified;
+ Ok(())
+ }
+
+ async fn cleanup_key_is_unique(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ key_column: &str,
+ ) -> Result<bool, Error> {
+ if self.verified_cleanup_keys.lock().await.contains(table) {
+ return Ok(true);
+ }
+ let check = cleanup_key_check(pool, table, key_column)
+ .await
+ .map_err(|e| {
+ Error::Connection(format!(
+ "failed to check primary_key_column '{key_column}' on table '{table}': {e}"
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ self.verified_cleanup_keys
+ .lock()
+ .await
+ .insert(table.to_string());
+ Ok(true)
+ }
+ CleanupKeyCheck::NotUnique => Ok(false),
+ CleanupKeyCheck::TableMissing => Err(Error::Connection(format!(
+ "table '{table}' disappeared before its primary_key_column could be checked"
+ ))),
+ }
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ /// Best current knowledge of how a table's tracking values compare.
+ async fn tracking_kind(&self, table: &str) -> OffsetKind {
+ self.tracking_kinds
+ .lock()
+ .await
+ .get(table)
+ .copied()
+ .unwrap_or(OffsetKind::Unknown)
+ }
+
+ /// Whether the stored cursor is what limits the rows a poll can see. The built
+ /// polling query always filters on it; a `custom_query` does so only if it
+ /// interpolates `$offset`.
+ fn cursor_gates_query(&self) -> bool {
+ match &self.config.custom_query {
+ Some(query) => query.contains("$offset"),
+ None => true,
+ }
+ }
+
+ /// The offset a poll actually resumes from: the stored cursor, or the configured
+ /// starting point before anything has been stored.
+ fn effective_offset(&self, last_offset: &Option<String>) -> Option<String> {
+ last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let offset_value = self.effective_offset(last_offset).unwrap_or_default();
+ let offset = format_offset_value(&offset_value, kind);
+ let quoted_table = quote_qualified_identifier(table)?;
+ let limit = batch_size.to_string();
+ let now = Utc::now();
+ let now_unix = now.timestamp().to_string();
+ let now = now.to_rfc3339();
+ let replacements = [
+ ("$table", quoted_table.as_str()),
+ ("$offset", offset.as_str()),
+ ("$limit", limit.as_str()),
+ ("$now_unix", now_unix.as_str()),
+ ("$now", now.as_str()),
+ ];
+
+ Ok(substitute_query_tokens(query, &replacements))
+ }
+
+ /// The query is substituted as text and re-executed once per configured table on
+ /// every poll for the life of the process, so a shape that cannot terminate does
+ /// not fail, it repeats. What can be judged from the text alone is judged here,
+ /// at startup, rather than left to the first poll.
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let tracking_column = self.tracking_column();
+
+ match leading_statement_token(query) {
+ None => {
+ return Err(Error::InitError(
+ "custom_query has no statement: it is empty, or nothing but comments"
+ .to_string(),
+ ));
+ }
+ Some(token) if !READ_STATEMENT_STARTS.contains(&token.as_str()) => {
+ return Err(Error::InitError(format!(
+ "custom_query must be a read starting with SELECT, WITH or TABLE, found \
+ '{token}'"
+ )));
+ }
+ Some(_) => {}
+ }
+
+ // Collapsed so `ORDER\nBY` and `ORDER BY` read the same as `ORDER BY`; the
+ // check hard-fails, so a formatting difference must not decide it.
+ let normalized = query
+ .split_whitespace()
+ .collect::<Vec<_>>()
+ .join(" ")
+ .to_uppercase();
+
+ if query.contains("$offset") {
+ if !normalized.contains("ORDER BY") {
+ return Err(Error::InitError(format!(
+ "custom_query filters on $offset but has no ORDER BY, add \
+ `ORDER BY {tracking_column}`"
+ )));
+ }
+ } else if !self.config.delete_after_read.unwrap_or(false)
+ && self.config.processed_column.is_none()
+ {
+ return Err(Error::InitError(format!(
+ "custom_query must filter with $offset (`WHERE {tracking_column} > $offset`), or \
+ set delete_after_read or processed_column"
+ )));
+ }
+
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "custom_query uses $table but no tables are configured".to_string(),
+ ));
+ }
+ // Without $table the same SQL is executed once per table, so identical rows
+ // would be emitted repeatedly, each carrying a different table's metadata,
+ // message ID, and tracking offset. Table identity is not recoverable from a
+ // static query, so reject the combination instead of guessing one.
+ if !query.contains("$table") && self.config.tables.len() > 1 {
+ return Err(Error::InitError(format!(
+ "custom_query has no $table placeholder but {} tables are configured ({}), use \
+ $table in the query or configure a single table",
+ self.config.tables.len(),
+ self.config.tables.join(", ")
+ )));
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = self.effective_offset(last_offset) {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(&offset, kind)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn apply_pending_operations(
+ &self,
+ operations: Vec<PendingOperation>,
+ deadline: tokio::time::Instant,
+ ) -> Result<Vec<PendingOperation>, Error> {
+ let mut failed_operations = Vec::new();
+ let mut operations = operations.into_iter();
+ while let Some(operation) = operations.next() {
+ if tokio::time::Instant::now() >= deadline {
+ failed_operations.push(operation);
+ failed_operations.extend(operations);
+ break;
+ }
+ let result = tokio::time::timeout_at(deadline, async {
+ let pool = self.get_pool()?;
+ self.mark_or_delete_processed_rows(pool, &operation).await
+ })
+ .await;
+ match result {
+ Ok(Ok(())) => {
+ self.consecutive_cleanup_failures
+ .store(0, Ordering::Relaxed);
+ }
+ Ok(Err(error)) => {
+ self.record_cleanup_failure(&operation, &error)?;
+ failed_operations.push(operation);
+ }
+ Err(_) => {
+ self.record_cleanup_failure(
+ &operation,
+ "cleanup exceeded the 10s budget for one Ack",
+ )?;
+ failed_operations.push(operation);
+ }
+ }
+ }
+ Ok(failed_operations)
+ }
+
+ fn record_cleanup_failure(
+ &self,
+ operation: &PendingOperation,
+ error: impl std::fmt::Display,
+ ) -> Result<(), Error> {
+ let consecutive_failures = self
+ .consecutive_cleanup_failures
+ .fetch_add(1, Ordering::Relaxed)
+ .saturating_add(1);
+ error!(
+ "Failed to mark or delete {} delivered row(s) in table '{}' for {CONNECTOR_NAME} \
+ connector ID: {}, retrying on the next poll ({consecutive_failures} consecutive \
+ failures): {error}",
+ operation.ids.len(),
+ operation.table,
+ self.id
+ );
+ if consecutive_failures >= MAX_CONSECUTIVE_CLEANUP_FAILURES {
+ return Err(Error::Connection(format!(
+ "stopping {CONNECTOR_NAME} connector ID {} after {consecutive_failures} \
+ consecutive mark/delete failures; the cleanup stays in the saved state and \
+ runs again on restart",
+ self.id
+ )));
+ }
+ Ok(())
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ operation: &PendingOperation,
+ ) -> Result<(), Error> {
+ let PendingOperation { table, ids, target } = operation;
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(&target.key_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = match &target.action {
+ CleanupAction::Delete => {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
Review Comment:
critical: cleanup replay matches only the primary key, so a crash can let an old operation delete an unread replacement row. save each row's tracking value with its key and match both during cleanup.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,4071 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source,
+ source::SourceBatchResult, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+/// Upper bound on applying a batch's staged mark/delete after its Ack. The SDK waits
+/// `BATCH_RESULT_TIMEOUT` (30s) for the result, so this has to finish well inside it.
+const ACK_BATCH_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// MySQL rejects a prepared statement with more placeholders than this (error 1390), and
+/// the mark/delete binds one per row of the batch.
+const MAX_CLEANUP_BATCH_SIZE: u32 = 65_535;
+
+/// How many mark/delete statements in a row may fail before the connector stops. The
+/// operations stay in the persisted state and run again on restart.
+const MAX_CONSECUTIVE_CLEANUP_FAILURES: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ pending_batch: Mutex<Option<PendingBatch>>,
+ consecutive_cleanup_failures: AtomicU32,
+ verified_cleanup_keys: Mutex<HashSet<String>>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some("it has too few distinct values to be unique per row");
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some("binary values are compared as base64 text, which does not keep byte order");
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some("MySQL orders these by declaration index, not by label");
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL \
+ or not an ordered scalar (tinyint(1), JSON)"
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "row '{value}' is below the stored offset '{previous}' for tracking_column \
+ '{column}'. Filter the query with `{column} > $offset`"
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. Order the query ascending by '{column}'"
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+ /// Mark/delete work for a batch the runtime has already sent. It is persisted with
+ /// the batch's offsets so a crash between the send and the cleanup reruns it.
+ #[serde(default)]
+ pending_operations: Vec<PendingOperation>,
+}
+
+/// The rows of one table to mark or delete once their batch is acknowledged.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct PendingOperation {
+ table: String,
+ ids: Vec<PkValue>,
+ target: CleanupTarget,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct CleanupTarget {
+ key_column: String,
+ action: CleanupAction,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum CleanupAction {
+ Delete,
+ MarkProcessed { column: String },
+}
+
+/// The candidate state of the batch most recently handed to the runtime. Live state
+/// only takes it over on Ack, so a Nack leaves offsets and rows where they were.
+#[derive(Debug, Clone)]
+struct PendingBatch {
+ state: State,
+ /// The runtime has delivered the batch but its cleanup has not finished yet.
+ acknowledged: bool,
+ /// Cleanup finished and the next poll must persist `state` without the operations,
+ /// so a restart does not run them again.
+ retiring: bool,
+}
+
+struct PolledBatch {
+ messages: Vec<ProducedMessage>,
+ pending: Option<PendingBatch>,
+}
+
+enum CleanupKeyCheck {
+ Unique,
+ NotUnique,
+ TableMissing,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty, list the table even when custom_query hardcodes it"
+ .to_string(),
+ ));
+ }
+
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ if let Some(batch_size) = self.config.batch_size
+ && batch_size > MAX_CLEANUP_BATCH_SIZE
+ && self.cleanup_target().is_some()
+ {
+ return Err(Error::InitError(format!(
+ "batch_size {batch_size} exceeds {MAX_CLEANUP_BATCH_SIZE} with delete_after_read or \
+ processed_column set, MySQL allows at most {MAX_CLEANUP_BATCH_SIZE} placeholders \
+ in the statement that marks or deletes a batch"
+ )));
+ }
+
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "name the column custom_query orders by and filters with $offset"
+ } else {
+ "use a unique, monotonically increasing column"
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set, {guidance}"
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.validate_pending_cleanup()?;
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+ self.validate_cleanup_keys().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ if let Some(state) = self.retiring_state().await {
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: Some(self.serialize_pending_state(&state)?),
+ });
+ }
+
+ // Skip the pacing delay while draining a backlog, but never while a cleanup
+ // retry is pending, or a failing mark/delete would spin without a pause.
+ let cleanup_pending = self.pending_batch.lock().await.is_some();
+ if cleanup_pending || !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ {
+ let pending = self.pending_batch.lock().await;
+ if let Some(pending) = pending.as_ref() {
+ if !pending.acknowledged {
+ error!(
+ "{CONNECTOR_NAME} connector ID: {} was polled while a batch was still in flight",
+ self.id
+ );
+ return Err(Error::InvalidState);
+ }
+ // An empty batch whose Ack retries the cleanup the last one left behind.
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: None,
+ });
+ }
+ }
+
+ let polled = self.poll_tables().await?;
+
+ let processed_rows = match &polled.pending {
+ Some(pending) => pending.state.processed_rows,
+ None => self.state.lock().await.processed_rows,
+ };
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ }
+
+ // Idle cycles stage nothing, so return None to let the runtime skip the fsync.
+ let persisted_state = polled
+ .pending
+ .as_ref()
+ .map(|pending| self.serialize_pending_state(&pending.state))
+ .transpose()?;
+ *self.pending_batch.lock().await = polled.pending;
+
+ Ok(ProducedMessages {
+ schema,
+ messages: polled.messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> {
+ if result == SourceBatchResult::Nack {
+ // Dropping the candidate leaves the live offsets where they were, so the
+ // next poll reads the same rows again. Delivered batches keep their cleanup.
+ let mut pending = self.pending_batch.lock().await;
+ if pending
+ .as_ref()
+ .is_some_and(|pending| !pending.acknowledged && !pending.retiring)
+ {
+ pending.take();
+ }
+ return Ok(());
+ }
+
+ let Some(PendingBatch { mut state, .. }) = self.pending_batch.lock().await.clone() else {
+ return Ok(());
+ };
+
+ let operations = std::mem::take(&mut state.pending_operations);
+ let had_operations = !operations.is_empty();
+ let deadline = tokio::time::Instant::now() + ACK_BATCH_TIMEOUT;
+ let failed_operations = self.apply_pending_operations(operations, deadline).await?;
+
+ if failed_operations.is_empty() {
+ *self.state.lock().await = state.clone();
+ let mut pending = self.pending_batch.lock().await;
+ if had_operations {
+ *pending = Some(PendingBatch {
+ state,
+ acknowledged: false,
+ retiring: true,
+ });
+ } else {
+ pending.take();
+ }
+ } else {
+ state.pending_operations = failed_operations;
+ *self.pending_batch.lock().await = Some(PendingBatch {
+ state,
+ acknowledged: true,
+ retiring: false,
+ });
+ }
+
+ Ok(())
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ let state = restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ pending_operations: Vec::new(),
+ });
+ // The runtime only persists a batch's operations after sending it, so restored
+ // operations belong to a delivered batch and only the cleanup is left to do.
+ let pending_batch = (!state.pending_operations.is_empty()).then(|| PendingBatch {
+ state: state.clone(),
+ acknowledged: true,
+ retiring: false,
+ });
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(state),
+ pending_batch: Mutex::new(pending_batch),
+ consecutive_cleanup_failures: AtomicU32::new(0),
+ verified_cleanup_keys: Mutex::new(HashSet::new()),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ poisoned_tables: Mutex::new(HashSet::new()),
+ tracking_kinds: Mutex::new(HashMap::new()),
+ polls: AtomicU64::new(0),
+ cursor_regressions: Mutex::new(HashMap::new()),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ // `format_offset_value` escapes backslashes, which under NO_BACKSLASH_ESCAPES
+ // would reach MySQL as literal characters and filter on the wrong cursor.
+ .after_connect(|connection, _| {
+ Box::pin(async move {
+ sqlx::query(
+ "SET SESSION sql_mode = REPLACE(@@SESSION.sql_mode, 'NO_BACKSLASH_ESCAPES', '')",
+ )
+ .execute(&mut *connection)
+ .await?;
+ Ok(())
+ })
+ })
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn tracking_column(&self) -> &str {
+ self.config.tracking_column.as_deref().unwrap_or("id")
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn resolve_tracking_columns(&self) -> Result<HashMap<String, OffsetKind>, Error> {
+ // Whether the connector owns the query, and can therefore hold the schema to
+ // what that query will do with it.
+ let strict = self.config.custom_query.is_none();
+ let pool = self.get_pool()?;
+ let tracking_column = self.tracking_column();
+ let mut kinds = HashMap::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ }
+
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ }
+
+ warn!(
+ "tracking_column '{tracking_column}' is not a column of table '{table}'; \
+ assuming custom_query projects it. Its comparison order is resolved from \
+ the first result set instead."
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ // The column's type survives a plain projection, so it is worth rejecting
+ // whatever the query looks like. A custom_query aliasing an expression onto
+ // a name the table also uses is the one false positive, hence the hint.
+ if let Some(reason) = unusable_tracking_type(&data_type) {
+ let aliasing_hint = if self.config.custom_query.is_some() {
+ " If custom_query aliases a different expression to this name, rename the \
+ alias so it does not collide with the column."
+ } else {
+ ""
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': {reason}. Use an integer, timestamp, or string \
+ column.{aliasing_hint}"
+ )));
+ }
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; a row \
+ whose value is NULL cannot produce a cursor and will fail that table's batch. \
+ Have custom_query exclude those rows, or declare the column NOT NULL."
+ );
+ }
+
+ if is_collation_ordered(&data_type) {
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': values must be unique and increase in its collation order \
+ (zero-pad numbers, prefer a _bin collation)"
+ );
+ }
+
+ kinds.insert(table.clone(), offset_kind_for_data_type(&data_type));
+ }
+ Ok(kinds)
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn serialize_pending_state(&self, state: &State) -> Result<ConnectorState, Error> {
+ self.serialize_state(state).ok_or_else(|| {
+ Error::Serialization(format!("failed to serialize {CONNECTOR_NAME} state"))
+ })
+ }
+
+ async fn retiring_state(&self) -> Option<State> {
+ self.pending_batch
+ .lock()
+ .await
+ .as_ref()
+ .filter(|pending| pending.retiring)
+ .map(|pending| pending.state.clone())
+ }
+
+ fn primary_key_column(&self) -> &str {
+ self.config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(self.tracking_column())
+ }
+
+ fn cleanup_target(&self) -> Option<CleanupTarget> {
+ let action = if self.config.delete_after_read.unwrap_or(false) {
+ CleanupAction::Delete
+ } else if let Some(column) = &self.config.processed_column {
+ CleanupAction::MarkProcessed {
+ column: column.clone(),
+ }
+ } else {
+ return None;
+ };
+ Some(CleanupTarget {
+ key_column: self.primary_key_column().to_string(),
+ action,
+ })
+ }
+
+ /// Restored operations were staged under the config of the run that wrote them.
+ /// Running them under a different key or action would touch rows nobody read.
+ fn validate_pending_cleanup(&mut self) -> Result<(), Error> {
+ let configured = self.cleanup_target();
+ for operation in &self.state.get_mut().pending_operations {
+ if configured.as_ref() != Some(&operation.target) {
+ return Err(Error::InitError(format!(
+ "the cleanup configuration changed while cleanup of {} delivered row(s) in \
+ table '{}' is pending; restore primary_key_column '{}' and action {:?}",
+ operation.ids.len(),
+ operation.table,
+ operation.target.key_column,
+ operation.target.action
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ /// `WHERE key IN (...)` on a key that repeats marks or deletes every row sharing a
+ /// value, including ones that were never read. Tables that don't exist yet are
+ /// checked by the first poll that stages cleanup for them.
+ async fn validate_cleanup_keys(&mut self) -> Result<(), Error> {
+ let Some(target) = self.cleanup_target() else {
+ return Ok(());
+ };
+ let pool = self.get_pool()?;
+ let mut verified = HashSet::with_capacity(self.config.tables.len());
+ for table in &self.config.tables {
+ let check = cleanup_key_check(pool, table, &target.key_column)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to check primary_key_column '{}' on table '{table}': {e}",
+ target.key_column
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ verified.insert(table.clone());
+ }
+ CleanupKeyCheck::NotUnique => {
+ return Err(Error::InitError(not_unique_key_message(
+ table,
+ &target.key_column,
+ )));
+ }
+ CleanupKeyCheck::TableMissing => {}
+ }
+ }
+ *self.verified_cleanup_keys.get_mut() = verified;
+ Ok(())
+ }
+
+ async fn cleanup_key_is_unique(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ key_column: &str,
+ ) -> Result<bool, Error> {
+ if self.verified_cleanup_keys.lock().await.contains(table) {
+ return Ok(true);
+ }
+ let check = cleanup_key_check(pool, table, key_column)
+ .await
+ .map_err(|e| {
+ Error::Connection(format!(
+ "failed to check primary_key_column '{key_column}' on table '{table}': {e}"
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ self.verified_cleanup_keys
+ .lock()
+ .await
+ .insert(table.to_string());
+ Ok(true)
+ }
+ CleanupKeyCheck::NotUnique => Ok(false),
+ CleanupKeyCheck::TableMissing => Err(Error::Connection(format!(
+ "table '{table}' disappeared before its primary_key_column could be checked"
+ ))),
+ }
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ /// Best current knowledge of how a table's tracking values compare.
+ async fn tracking_kind(&self, table: &str) -> OffsetKind {
+ self.tracking_kinds
+ .lock()
+ .await
+ .get(table)
+ .copied()
+ .unwrap_or(OffsetKind::Unknown)
+ }
+
+ /// Whether the stored cursor is what limits the rows a poll can see. The built
+ /// polling query always filters on it; a `custom_query` does so only if it
+ /// interpolates `$offset`.
+ fn cursor_gates_query(&self) -> bool {
+ match &self.config.custom_query {
+ Some(query) => query.contains("$offset"),
+ None => true,
+ }
+ }
+
+ /// The offset a poll actually resumes from: the stored cursor, or the configured
+ /// starting point before anything has been stored.
+ fn effective_offset(&self, last_offset: &Option<String>) -> Option<String> {
+ last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let offset_value = self.effective_offset(last_offset).unwrap_or_default();
+ let offset = format_offset_value(&offset_value, kind);
+ let quoted_table = quote_qualified_identifier(table)?;
+ let limit = batch_size.to_string();
+ let now = Utc::now();
+ let now_unix = now.timestamp().to_string();
+ let now = now.to_rfc3339();
+ let replacements = [
+ ("$table", quoted_table.as_str()),
+ ("$offset", offset.as_str()),
+ ("$limit", limit.as_str()),
+ ("$now_unix", now_unix.as_str()),
+ ("$now", now.as_str()),
+ ];
+
+ Ok(substitute_query_tokens(query, &replacements))
+ }
+
+ /// The query is substituted as text and re-executed once per configured table on
+ /// every poll for the life of the process, so a shape that cannot terminate does
+ /// not fail, it repeats. What can be judged from the text alone is judged here,
+ /// at startup, rather than left to the first poll.
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let tracking_column = self.tracking_column();
+
+ match leading_statement_token(query) {
+ None => {
+ return Err(Error::InitError(
+ "custom_query has no statement: it is empty, or nothing but comments"
+ .to_string(),
+ ));
+ }
+ Some(token) if !READ_STATEMENT_STARTS.contains(&token.as_str()) => {
+ return Err(Error::InitError(format!(
+ "custom_query must be a read starting with SELECT, WITH or TABLE, found \
+ '{token}'"
+ )));
+ }
+ Some(_) => {}
+ }
+
+ // Collapsed so `ORDER\nBY` and `ORDER BY` read the same as `ORDER BY`; the
+ // check hard-fails, so a formatting difference must not decide it.
+ let normalized = query
+ .split_whitespace()
+ .collect::<Vec<_>>()
+ .join(" ")
+ .to_uppercase();
+
+ if query.contains("$offset") {
+ if !normalized.contains("ORDER BY") {
+ return Err(Error::InitError(format!(
+ "custom_query filters on $offset but has no ORDER BY, add \
+ `ORDER BY {tracking_column}`"
+ )));
+ }
+ } else if !self.config.delete_after_read.unwrap_or(false)
+ && self.config.processed_column.is_none()
+ {
+ return Err(Error::InitError(format!(
+ "custom_query must filter with $offset (`WHERE {tracking_column} > $offset`), or \
+ set delete_after_read or processed_column"
+ )));
+ }
+
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "custom_query uses $table but no tables are configured".to_string(),
+ ));
+ }
+ // Without $table the same SQL is executed once per table, so identical rows
+ // would be emitted repeatedly, each carrying a different table's metadata,
+ // message ID, and tracking offset. Table identity is not recoverable from a
+ // static query, so reject the combination instead of guessing one.
+ if !query.contains("$table") && self.config.tables.len() > 1 {
+ return Err(Error::InitError(format!(
+ "custom_query has no $table placeholder but {} tables are configured ({}), use \
+ $table in the query or configure a single table",
+ self.config.tables.len(),
+ self.config.tables.join(", ")
+ )));
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = self.effective_offset(last_offset) {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(&offset, kind)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn apply_pending_operations(
+ &self,
+ operations: Vec<PendingOperation>,
+ deadline: tokio::time::Instant,
+ ) -> Result<Vec<PendingOperation>, Error> {
+ let mut failed_operations = Vec::new();
+ let mut operations = operations.into_iter();
+ while let Some(operation) = operations.next() {
+ if tokio::time::Instant::now() >= deadline {
+ failed_operations.push(operation);
+ failed_operations.extend(operations);
+ break;
+ }
+ let result = tokio::time::timeout_at(deadline, async {
+ let pool = self.get_pool()?;
+ self.mark_or_delete_processed_rows(pool, &operation).await
+ })
+ .await;
+ match result {
+ Ok(Ok(())) => {
+ self.consecutive_cleanup_failures
+ .store(0, Ordering::Relaxed);
+ }
+ Ok(Err(error)) => {
+ self.record_cleanup_failure(&operation, &error)?;
+ failed_operations.push(operation);
+ }
+ Err(_) => {
+ self.record_cleanup_failure(
+ &operation,
+ "cleanup exceeded the 10s budget for one Ack",
+ )?;
+ failed_operations.push(operation);
+ }
+ }
+ }
+ Ok(failed_operations)
+ }
+
+ fn record_cleanup_failure(
+ &self,
+ operation: &PendingOperation,
+ error: impl std::fmt::Display,
+ ) -> Result<(), Error> {
+ let consecutive_failures = self
+ .consecutive_cleanup_failures
+ .fetch_add(1, Ordering::Relaxed)
+ .saturating_add(1);
+ error!(
+ "Failed to mark or delete {} delivered row(s) in table '{}' for {CONNECTOR_NAME} \
+ connector ID: {}, retrying on the next poll ({consecutive_failures} consecutive \
+ failures): {error}",
+ operation.ids.len(),
+ operation.table,
+ self.id
+ );
+ if consecutive_failures >= MAX_CONSECUTIVE_CLEANUP_FAILURES {
+ return Err(Error::Connection(format!(
+ "stopping {CONNECTOR_NAME} connector ID {} after {consecutive_failures} \
+ consecutive mark/delete failures; the cleanup stays in the saved state and \
+ runs again on restart",
+ self.id
+ )));
+ }
+ Ok(())
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ operation: &PendingOperation,
+ ) -> Result<(), Error> {
+ let PendingOperation { table, ids, target } = operation;
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(&target.key_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = match &target.action {
+ CleanupAction::Delete => {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
+ }
+ CleanupAction::MarkProcessed { column } => {
+ let quoted_processed = quote_identifier(column)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({placeholders})"
+ )
+ }
+ };
+
+ // The query and its binds are rebuilt on every attempt because with_retry
+ // calls the closure once per try and each `execute` consumes the query.
+ with_retry(
+ || {
+ let mut prepared = sqlx::query(sqlx::AssertSqlSafe(query.as_str()));
+ for id in ids {
+ prepared = match id {
+ PkValue::Text(text) => prepared.bind(text),
+ PkValue::Bytes(bytes) => prepared.bind(bytes),
+ };
+ }
+ prepared.execute(pool)
+ },
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<PolledBatch, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.tracking_column();
+ let pk_column = self.primary_key_column();
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ //
+ // An ordering violation is handled one step further. Retrying it would
+ // re-run the same query, re-decode the same rows, and hit the same failure
+ // forever, so the table is latched off instead of being retried each cycle.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ let poll_number = self.polls.fetch_add(1, Ordering::Relaxed) + 1;
+ let poisoned = { self.poisoned_tables.lock().await.clone() };
+ let mut newly_poisoned: Vec<String> = Vec::new();
+
+ for table in &self.config.tables {
+ if poisoned.contains(table) {
+ continue;
+ }
+
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ if !batch.messages.is_empty() {
+ self.cursor_regressions.lock().await.remove(table);
+ }
+ // A full batch means the LIMIT was hit, so more rows are waiting.
+ any_batch_full |= batch.messages.len() as u32 >= batch_size;
+ batches.push(batch);
+ }
+ Err(FetchError::Ordering(reason) | FetchError::MissingTrackingColumn(reason)) => {
+ error!(
+ "Table '{table}' at offset {}: {reason}. Table disabled until restart.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ newly_poisoned.push(table.clone());
+ }
+ Err(FetchError::CursorRegression(reason)) => {
+ // Cleared only by a batch that carried rows, so the run counts polls
+ // since the table last published anything. An idle poll produces as
+ // little as a rejected one does and must not read as recovery.
+ let consecutive = {
+ let mut regressions = self.cursor_regressions.lock().await;
+ let consecutive = regressions.entry(table.clone()).or_insert(0);
+ *consecutive += 1;
+ *consecutive
+ };
+ let offset = last_offset.as_deref().unwrap_or("<start>");
+ if consecutive >= CURSOR_REGRESSION_ERROR_THRESHOLD {
+ error!(
+ "Table '{table}' at offset {offset}: {reason}. Skipping this poll \
+ ({consecutive} consecutive polls)."
+ );
+ } else {
+ warn!("Table '{table}' at offset {offset}: {reason}. Skipping this poll.");
+ }
+ }
+ Err(FetchError::StaleOffsetKind(reason)) => {
+ warn!(
+ "Table '{table}' at offset {}: {reason}. Retrying on the next poll.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::UnusableTrackingValue(reason)) => {
+ error!(
+ "Table '{table}' at offset {}: {reason}. Retrying each poll until the row \
+ is fixed.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::Other(error)) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ if !newly_poisoned.is_empty() {
+ let mut poisoned_tables = self.poisoned_tables.lock().await;
+ poisoned_tables.extend(newly_poisoned);
+ if self
+ .config
+ .tables
+ .iter()
+ .all(|table| poisoned_tables.contains(table))
+ {
+ error!(
+ "All configured tables are disabled because their batches cannot yield a \
+ cursor; {CONNECTOR_NAME} connector with ID: {} is idle until restarted",
+ self.id
+ );
+ }
+ }
+
+ // From the next poll on a disabled table is skipped before anything is logged,
+ // so the one line written when it was disabled can scroll out of the operator's
+ // log window and leave a dark connector looking healthy.
+ if poll_number.is_multiple_of(DISABLED_TABLES_LOG_INTERVAL) {
+ let poisoned_tables = self.poisoned_tables.lock().await;
+ if !poisoned_tables.is_empty() {
+ let mut disabled: Vec<&str> = poisoned_tables.iter().map(String::as_str).collect();
+ disabled.sort_unstable();
+ error!(
+ "{} of {} configured table(s) are still disabled and are not being polled by \
+ {CONNECTOR_NAME} connector with ID: {}, and will stay that way until it is \
+ restarted: {}",
+ disabled.len(),
+ self.config.tables.len(),
+ self.id,
+ disabled.join(", ")
+ );
+ }
+ }
+
+ // Phase 2: stage each table's batch. Nothing touches MySQL or the live state
+ // here: the offsets and the mark/delete are applied in `on_batch_result` once
+ // the runtime has sent the batch and saved this candidate state, and a Nack
+ // throws the candidate away so the same rows are read again.
+ let cleanup_target = self.cleanup_target();
+ let mut candidate_state = self.state.lock().await.clone();
+ let mut operations = Vec::new();
+ let mut messages = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if let Some(target) = &cleanup_target
+ && !batch.messages.is_empty()
+ {
+ if batch.processed_ids.is_empty() {
+ error!(
+ "Table '{}': mark/delete is configured but no primary keys were extracted \
+ from {} row(s), skipping this cycle so rows are not published without \
+ being marked or deleted (check that '{pk_column}' is projected and scalar)",
+ batch.table,
+ batch.messages.len()
+ );
+ continue;
+ }
+
+ match self
+ .cleanup_key_is_unique(pool, &batch.table, &target.key_column)
+ .await
+ {
+ Ok(true) => {}
+ Ok(false) => {
+ error!(
+ "{}. Table disabled until restart.",
+ not_unique_key_message(&batch.table, &target.key_column)
+ );
+ self.poisoned_tables.lock().await.insert(batch.table);
+ continue;
+ }
+ Err(error) => {
+ error!("Table '{}' skipped this cycle: {error}", batch.table);
+ continue;
+ }
+ }
+
+ operations.push(PendingOperation {
+ table: batch.table.clone(),
+ ids: std::mem::take(&mut batch.processed_ids),
+ target: target.clone(),
+ });
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ candidate_state.tracking_offsets.insert(batch.table, offset);
+ }
+ }
+
+ let pending = (total_processed > 0).then(|| {
+ candidate_state.processed_rows += total_processed;
+ candidate_state.pending_operations = operations;
+ PendingBatch {
+ state: candidate_state,
+ acknowledged: false,
+ retiring: false,
+ }
+ });
+
+ self.last_batch_full
+ .store(any_batch_full, Ordering::Relaxed);
+ Ok(PolledBatch { messages, pending })
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, FetchError> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query_kind = self.tracking_kind(table).await;
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.substitute_query_params(custom_query, table, last_offset, batch_size, query_kind)?
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size, query_kind)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ // Settled once for the whole batch rather than per row, since the column list
+ // is a property of the result set.
+ if let Some(first) = rows.first() {
+ ensure_tracking_column_projected(
+ first.columns().iter().map(|column| column.name()),
+ tracking_column,
+ )?;
+ }
+
+ let resume_from = if self.cursor_gates_query() {
+ self.effective_offset(last_offset)
+ } else {
+ None
+ };
+
+ // The result set describes what this query actually returned, so it outranks
+ // whatever `information_schema` said at startup and is the only answer
+ // available for a computed column.
+ let kind = match rows.first().and_then(|row| {
+ observed_tracking_kind(row, tracking_column).filter(|observed| *observed != query_kind)
+ }) {
+ Some(observed) => {
+ self.tracking_kinds
+ .lock()
+ .await
+ .insert(table.to_string(), observed);
+ // The query was already sent with the stale kind, so its offset literal
+ // may be quoted for the wrong type.
+ if let Some(offset) = &resume_from
+ && offset_literal_diverged(offset, query_kind, observed)
+ {
+ return Err(FetchError::StaleOffsetKind(format!(
+ "tracking_column '{tracking_column}' type differs from the one detected \
+ at startup (filtered with {}, expected {}), batch discarded",
+ format_offset_value(offset, query_kind),
+ format_offset_value(offset, observed)
+ )));
+ }
+ observed
+ }
+ None => query_kind,
+ };
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ // Every row of a result set shares its columns, so the published key names are
+ // derived once per batch instead of once per row.
+ let column_names: Vec<String> = rows
+ .first()
+ .map(|row| {
+ row.columns()
+ .iter()
+ .map(|column| {
+ if table_config.snake_case_columns {
+ to_snake_case(column.name())
+ } else {
+ column.name().to_string()
+ }
+ })
+ .collect()
+ })
+ .unwrap_or_default();
+ let mut ordering = OrderingGuard::new(tracking_column, kind, resume_from);
+ for row in rows {
+ let processed = self
+ .process_row(&row, &table_config, &column_names)
+ .map_err(|e| {
+ error!(
+ "Failed to decode row in table '{table}' (pk {}): {e}",
+ pk_for_log(&row, table_config.pk_column)
+ );
+ e
+ })?;
+
+ let Some(value) = processed.tracking_value else {
+ return Err(unusable_tracking_value(
+ tracking_column,
+ &pk_for_log(&row, table_config.pk_column),
+ ));
+ };
+ ordering
+ .accept(value)
+ .map_err(|violation| match violation {
+ OrderingViolation::WithinBatch(reason) => FetchError::Ordering(reason),
+ OrderingViolation::BelowCursor(reason) => FetchError::CursorRegression(reason),
+ })?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+
+ batch.messages.push(processed.message);
+ }
+ batch.max_offset = ordering.into_cursor();
+
+ Ok(batch)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ column_names: &[String],
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<PkValue> = None;
+ let mut tracking_value: Option<String> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
+ let mut extracted_payload: Option<Vec<u8>> = None;
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ if name.eq_ignore_ascii_case(config.payload_col) {
+ extracted_payload =
+ Some(self.extract_payload_column(row, i, config.payload_format)?);
+ }
+ let is_tracking = name.eq_ignore_ascii_case(config.tracking_column);
+ let is_pk = name.eq_ignore_ascii_case(config.pk_column);
+ if is_tracking {
+ let value = extract_column_value(row, i)?;
+ tracking_value = column_value_as_string(row, i, &value)?;
+ if is_pk {
+ row_pk = pk_value_from_decoded(row, i, &value)?;
+ }
+ } else if is_pk {
+ row_pk = extract_pk_value(row, i)?;
+ }
+ }
+
+ // Every extract_payload_column arm yields Ok for a NULL column, so None
+ // here means the column was absent. Falling back to the whole-row JSON
+ // would publish bytes contradicting the schema derived from payload_format.
+ let payload = extracted_payload.ok_or_else(|| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': payload_column '{}' not present in the result set",
+ config.table, config.payload_col
+ ))
+ })?;
+
+ return Ok(build_processed_row(
+ config.table,
+ payload,
+ tracking_value,
+ row_pk,
+ ));
+ }
+
+ let mut data = serde_json::Map::new();
+ for ((i, column), column_name) in row.columns().iter().enumerate().zip(column_names) {
+ let name = column.name();
+ let value = extract_column_value(row, i)?;
+ if name.eq_ignore_ascii_case(config.tracking_column) {
+ tracking_value = column_value_as_string(row, i, &value)?;
+ }
+ if name.eq_ignore_ascii_case(config.pk_column) {
+ row_pk = pk_value_from_decoded(row, i, &value)?;
+ }
+ data.insert(column_name.clone(), value);
+ }
+
+ let payload = if config.include_metadata {
+ let record = DatabaseRecord {
+ table_name: config.table.to_string(),
+ operation_type: "SELECT".to_string(),
+ timestamp: Utc::now(),
+ data: serde_json::Value::Object(data),
+ old_data: None,
+ };
+ simd_json::to_vec(&record).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': failed to serialize row to JSON: {e}",
+ config.table
+ ))
+ })?
+ } else {
+ simd_json::to_vec(&data).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': failed to serialize row to JSON: {e}",
+ config.table
+ ))
+ })?
+ };
+
+ Ok(build_processed_row(
+ config.table,
+ payload,
+ tracking_value,
+ row_pk,
+ ))
+ }
+}
+
+/// Type-faithful primary-key extraction. Binary columns are taken as raw bytes
+/// rather than routed through the base64 path in `extract_column_value`, so the
+/// value can be bound back into `WHERE pk IN (...)` and match the row. Everything
+/// else takes the string form MySQL parses back to the same value. Returns None
+/// only for a NULL pk, which a real primary key cannot be.
+fn extract_pk_value(row: &MySqlRow, column_index: usize) -> Result<Option<PkValue>, Error> {
+ if is_binary_column(row, column_index) {
+ return binary_pk_value(row, column_index);
+ }
+ let value = extract_column_value(row, column_index)?;
+ pk_value_from_decoded(row, column_index, &value)
+}
+
+/// Like `extract_pk_value` for a column the caller has already decoded, so the
+/// common key column is not decoded a second time per row.
+fn pk_value_from_decoded(
+ row: &MySqlRow,
+ column_index: usize,
+ value: &serde_json::Value,
+) -> Result<Option<PkValue>, Error> {
+ if is_binary_column(row, column_index) {
+ return binary_pk_value(row, column_index);
+ }
+ Ok(column_value_as_string(row, column_index, value)?.map(PkValue::Text))
+}
+
+fn is_binary_column(row: &MySqlRow, column_index: usize) -> bool {
+ matches!(
+ row.columns()[column_index].type_info().name(),
+ "BINARY" | "VARBINARY" | "TINYBLOB" | "BLOB" | "MEDIUMBLOB" | "LONGBLOB"
+ )
+}
+
+fn binary_pk_value(row: &MySqlRow, column_index: usize) -> Result<Option<PkValue>, Error> {
+ let column = &row.columns()[column_index];
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "primary key column '{}' (MySQL type '{}'): {e}",
+ column.name(),
+ column.type_info().name()
+ ))
+ })?;
+ Ok(bytes.map(PkValue::Bytes))
+}
+
+fn extract_column_value(row: &MySqlRow, column_index: usize) -> Result<serde_json::Value, Error> {
+ let column = &row.columns()[column_index];
+ let type_name = column.type_info().name();
+
+ let to_err = |e: sqlx::Error| {
+ Error::InvalidRecordValue(format!(
+ "column '{}' (MySQL type '{type_name}'): {e}",
+ column.name()
+ ))
+ };
+
+ match type_name {
+ "BOOLEAN" => {
+ let value: Option<bool> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::Bool)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT" => {
+ let value: Option<i8> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT" => {
+ let value: Option<i16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT" | "INT" => {
+ let value: Option<i32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT" => {
+ let value: Option<i64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT UNSIGNED" => {
+ let value: Option<u8> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT UNSIGNED" => {
+ let value: Option<u16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT UNSIGNED" | "INT UNSIGNED" => {
+ let value: Option<u32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT UNSIGNED" => {
+ let value: Option<u64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "FLOAT" => {
+ let value: Option<f32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as f64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DOUBLE" => {
+ let value: Option<f64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DECIMAL" => {
+ let value: Option<rust_decimal::Decimal> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|d| serde_json::Value::String(d.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIT" => {
+ let value: Option<u64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "YEAR" => {
+ let value: Option<u16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATE" => {
+ let value: Option<NaiveDate> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|d| serde_json::Value::String(d.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TIME" => {
+ let value: Option<NaiveTime> = row.try_get(column_index).map_err(to_err)?;
Review Comment:
critical: decoding `TIME` as `NaiveTime` drops negative signs and rejects valid intervals of 24 hours or more. use `MySqlTime` to preserve the sign and full hour range.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,4071 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source,
+ source::SourceBatchResult, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+/// Upper bound on applying a batch's staged mark/delete after its Ack. The SDK waits
+/// `BATCH_RESULT_TIMEOUT` (30s) for the result, so this has to finish well inside it.
+const ACK_BATCH_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// MySQL rejects a prepared statement with more placeholders than this (error 1390), and
+/// the mark/delete binds one per row of the batch.
+const MAX_CLEANUP_BATCH_SIZE: u32 = 65_535;
+
+/// How many mark/delete statements in a row may fail before the connector stops. The
+/// operations stay in the persisted state and run again on restart.
+const MAX_CONSECUTIVE_CLEANUP_FAILURES: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ pending_batch: Mutex<Option<PendingBatch>>,
+ consecutive_cleanup_failures: AtomicU32,
+ verified_cleanup_keys: Mutex<HashSet<String>>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some("it has too few distinct values to be unique per row");
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some("binary values are compared as base64 text, which does not keep byte order");
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some("MySQL orders these by declaration index, not by label");
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL \
+ or not an ordered scalar (tinyint(1), JSON)"
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "row '{value}' is below the stored offset '{previous}' for tracking_column \
+ '{column}'. Filter the query with `{column} > $offset`"
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. Order the query ascending by '{column}'"
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+ /// Mark/delete work for a batch the runtime has already sent. It is persisted with
+ /// the batch's offsets so a crash between the send and the cleanup reruns it.
+ #[serde(default)]
+ pending_operations: Vec<PendingOperation>,
+}
+
+/// The rows of one table to mark or delete once their batch is acknowledged.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct PendingOperation {
+ table: String,
+ ids: Vec<PkValue>,
+ target: CleanupTarget,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct CleanupTarget {
+ key_column: String,
+ action: CleanupAction,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum CleanupAction {
+ Delete,
+ MarkProcessed { column: String },
+}
+
+/// The candidate state of the batch most recently handed to the runtime. Live state
+/// only takes it over on Ack, so a Nack leaves offsets and rows where they were.
+#[derive(Debug, Clone)]
+struct PendingBatch {
+ state: State,
+ /// The runtime has delivered the batch but its cleanup has not finished yet.
+ acknowledged: bool,
+ /// Cleanup finished and the next poll must persist `state` without the operations,
+ /// so a restart does not run them again.
+ retiring: bool,
+}
+
+struct PolledBatch {
+ messages: Vec<ProducedMessage>,
+ pending: Option<PendingBatch>,
+}
+
+enum CleanupKeyCheck {
+ Unique,
+ NotUnique,
+ TableMissing,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty, list the table even when custom_query hardcodes it"
+ .to_string(),
+ ));
+ }
+
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ if let Some(batch_size) = self.config.batch_size
+ && batch_size > MAX_CLEANUP_BATCH_SIZE
+ && self.cleanup_target().is_some()
+ {
+ return Err(Error::InitError(format!(
+ "batch_size {batch_size} exceeds {MAX_CLEANUP_BATCH_SIZE} with delete_after_read or \
+ processed_column set, MySQL allows at most {MAX_CLEANUP_BATCH_SIZE} placeholders \
+ in the statement that marks or deletes a batch"
+ )));
+ }
+
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "name the column custom_query orders by and filters with $offset"
+ } else {
+ "use a unique, monotonically increasing column"
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set, {guidance}"
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.validate_pending_cleanup()?;
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+ self.validate_cleanup_keys().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ if let Some(state) = self.retiring_state().await {
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: Some(self.serialize_pending_state(&state)?),
+ });
+ }
+
+ // Skip the pacing delay while draining a backlog, but never while a cleanup
+ // retry is pending, or a failing mark/delete would spin without a pause.
+ let cleanup_pending = self.pending_batch.lock().await.is_some();
+ if cleanup_pending || !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ {
+ let pending = self.pending_batch.lock().await;
+ if let Some(pending) = pending.as_ref() {
+ if !pending.acknowledged {
+ error!(
+ "{CONNECTOR_NAME} connector ID: {} was polled while a batch was still in flight",
+ self.id
+ );
+ return Err(Error::InvalidState);
+ }
+ // An empty batch whose Ack retries the cleanup the last one left behind.
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: None,
+ });
+ }
+ }
+
+ let polled = self.poll_tables().await?;
+
+ let processed_rows = match &polled.pending {
+ Some(pending) => pending.state.processed_rows,
+ None => self.state.lock().await.processed_rows,
+ };
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ }
+
+ // Idle cycles stage nothing, so return None to let the runtime skip the fsync.
+ let persisted_state = polled
+ .pending
+ .as_ref()
+ .map(|pending| self.serialize_pending_state(&pending.state))
+ .transpose()?;
+ *self.pending_batch.lock().await = polled.pending;
+
+ Ok(ProducedMessages {
+ schema,
+ messages: polled.messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> {
+ if result == SourceBatchResult::Nack {
+ // Dropping the candidate leaves the live offsets where they were, so the
+ // next poll reads the same rows again. Delivered batches keep their cleanup.
+ let mut pending = self.pending_batch.lock().await;
+ if pending
+ .as_ref()
+ .is_some_and(|pending| !pending.acknowledged && !pending.retiring)
+ {
+ pending.take();
+ }
+ return Ok(());
+ }
+
+ let Some(PendingBatch { mut state, .. }) = self.pending_batch.lock().await.clone() else {
+ return Ok(());
+ };
+
+ let operations = std::mem::take(&mut state.pending_operations);
+ let had_operations = !operations.is_empty();
+ let deadline = tokio::time::Instant::now() + ACK_BATCH_TIMEOUT;
+ let failed_operations = self.apply_pending_operations(operations, deadline).await?;
+
+ if failed_operations.is_empty() {
+ *self.state.lock().await = state.clone();
+ let mut pending = self.pending_batch.lock().await;
+ if had_operations {
+ *pending = Some(PendingBatch {
+ state,
+ acknowledged: false,
+ retiring: true,
+ });
+ } else {
+ pending.take();
+ }
+ } else {
+ state.pending_operations = failed_operations;
+ *self.pending_batch.lock().await = Some(PendingBatch {
+ state,
+ acknowledged: true,
+ retiring: false,
+ });
+ }
+
+ Ok(())
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ let state = restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ pending_operations: Vec::new(),
+ });
+ // The runtime only persists a batch's operations after sending it, so restored
+ // operations belong to a delivered batch and only the cleanup is left to do.
+ let pending_batch = (!state.pending_operations.is_empty()).then(|| PendingBatch {
+ state: state.clone(),
+ acknowledged: true,
+ retiring: false,
+ });
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(state),
+ pending_batch: Mutex::new(pending_batch),
+ consecutive_cleanup_failures: AtomicU32::new(0),
+ verified_cleanup_keys: Mutex::new(HashSet::new()),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ poisoned_tables: Mutex::new(HashSet::new()),
+ tracking_kinds: Mutex::new(HashMap::new()),
+ polls: AtomicU64::new(0),
+ cursor_regressions: Mutex::new(HashMap::new()),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ // `format_offset_value` escapes backslashes, which under NO_BACKSLASH_ESCAPES
+ // would reach MySQL as literal characters and filter on the wrong cursor.
+ .after_connect(|connection, _| {
+ Box::pin(async move {
+ sqlx::query(
+ "SET SESSION sql_mode = REPLACE(@@SESSION.sql_mode, 'NO_BACKSLASH_ESCAPES', '')",
+ )
+ .execute(&mut *connection)
+ .await?;
+ Ok(())
+ })
+ })
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn tracking_column(&self) -> &str {
+ self.config.tracking_column.as_deref().unwrap_or("id")
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn resolve_tracking_columns(&self) -> Result<HashMap<String, OffsetKind>, Error> {
+ // Whether the connector owns the query, and can therefore hold the schema to
+ // what that query will do with it.
+ let strict = self.config.custom_query.is_none();
+ let pool = self.get_pool()?;
+ let tracking_column = self.tracking_column();
+ let mut kinds = HashMap::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ }
+
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ }
+
+ warn!(
+ "tracking_column '{tracking_column}' is not a column of table '{table}'; \
+ assuming custom_query projects it. Its comparison order is resolved from \
+ the first result set instead."
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ // The column's type survives a plain projection, so it is worth rejecting
+ // whatever the query looks like. A custom_query aliasing an expression onto
+ // a name the table also uses is the one false positive, hence the hint.
+ if let Some(reason) = unusable_tracking_type(&data_type) {
+ let aliasing_hint = if self.config.custom_query.is_some() {
+ " If custom_query aliases a different expression to this name, rename the \
+ alias so it does not collide with the column."
+ } else {
+ ""
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': {reason}. Use an integer, timestamp, or string \
+ column.{aliasing_hint}"
+ )));
+ }
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; a row \
+ whose value is NULL cannot produce a cursor and will fail that table's batch. \
+ Have custom_query exclude those rows, or declare the column NOT NULL."
+ );
+ }
+
+ if is_collation_ordered(&data_type) {
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': values must be unique and increase in its collation order \
+ (zero-pad numbers, prefer a _bin collation)"
+ );
+ }
+
+ kinds.insert(table.clone(), offset_kind_for_data_type(&data_type));
+ }
+ Ok(kinds)
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn serialize_pending_state(&self, state: &State) -> Result<ConnectorState, Error> {
+ self.serialize_state(state).ok_or_else(|| {
+ Error::Serialization(format!("failed to serialize {CONNECTOR_NAME} state"))
+ })
+ }
+
+ async fn retiring_state(&self) -> Option<State> {
+ self.pending_batch
+ .lock()
+ .await
+ .as_ref()
+ .filter(|pending| pending.retiring)
+ .map(|pending| pending.state.clone())
+ }
+
+ fn primary_key_column(&self) -> &str {
+ self.config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(self.tracking_column())
+ }
+
+ fn cleanup_target(&self) -> Option<CleanupTarget> {
+ let action = if self.config.delete_after_read.unwrap_or(false) {
+ CleanupAction::Delete
+ } else if let Some(column) = &self.config.processed_column {
+ CleanupAction::MarkProcessed {
+ column: column.clone(),
+ }
+ } else {
+ return None;
+ };
+ Some(CleanupTarget {
+ key_column: self.primary_key_column().to_string(),
+ action,
+ })
+ }
+
+ /// Restored operations were staged under the config of the run that wrote them.
+ /// Running them under a different key or action would touch rows nobody read.
+ fn validate_pending_cleanup(&mut self) -> Result<(), Error> {
+ let configured = self.cleanup_target();
+ for operation in &self.state.get_mut().pending_operations {
+ if configured.as_ref() != Some(&operation.target) {
+ return Err(Error::InitError(format!(
+ "the cleanup configuration changed while cleanup of {} delivered row(s) in \
+ table '{}' is pending; restore primary_key_column '{}' and action {:?}",
+ operation.ids.len(),
+ operation.table,
+ operation.target.key_column,
+ operation.target.action
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ /// `WHERE key IN (...)` on a key that repeats marks or deletes every row sharing a
+ /// value, including ones that were never read. Tables that don't exist yet are
+ /// checked by the first poll that stages cleanup for them.
+ async fn validate_cleanup_keys(&mut self) -> Result<(), Error> {
+ let Some(target) = self.cleanup_target() else {
+ return Ok(());
+ };
+ let pool = self.get_pool()?;
+ let mut verified = HashSet::with_capacity(self.config.tables.len());
+ for table in &self.config.tables {
+ let check = cleanup_key_check(pool, table, &target.key_column)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to check primary_key_column '{}' on table '{table}': {e}",
+ target.key_column
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ verified.insert(table.clone());
+ }
+ CleanupKeyCheck::NotUnique => {
+ return Err(Error::InitError(not_unique_key_message(
+ table,
+ &target.key_column,
+ )));
+ }
+ CleanupKeyCheck::TableMissing => {}
+ }
+ }
+ *self.verified_cleanup_keys.get_mut() = verified;
+ Ok(())
+ }
+
+ async fn cleanup_key_is_unique(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ key_column: &str,
+ ) -> Result<bool, Error> {
+ if self.verified_cleanup_keys.lock().await.contains(table) {
+ return Ok(true);
+ }
+ let check = cleanup_key_check(pool, table, key_column)
+ .await
+ .map_err(|e| {
+ Error::Connection(format!(
+ "failed to check primary_key_column '{key_column}' on table '{table}': {e}"
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ self.verified_cleanup_keys
+ .lock()
+ .await
+ .insert(table.to_string());
+ Ok(true)
+ }
+ CleanupKeyCheck::NotUnique => Ok(false),
+ CleanupKeyCheck::TableMissing => Err(Error::Connection(format!(
+ "table '{table}' disappeared before its primary_key_column could be checked"
+ ))),
+ }
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ /// Best current knowledge of how a table's tracking values compare.
+ async fn tracking_kind(&self, table: &str) -> OffsetKind {
+ self.tracking_kinds
+ .lock()
+ .await
+ .get(table)
+ .copied()
+ .unwrap_or(OffsetKind::Unknown)
+ }
+
+ /// Whether the stored cursor is what limits the rows a poll can see. The built
+ /// polling query always filters on it; a `custom_query` does so only if it
+ /// interpolates `$offset`.
+ fn cursor_gates_query(&self) -> bool {
+ match &self.config.custom_query {
+ Some(query) => query.contains("$offset"),
+ None => true,
+ }
+ }
+
+ /// The offset a poll actually resumes from: the stored cursor, or the configured
+ /// starting point before anything has been stored.
+ fn effective_offset(&self, last_offset: &Option<String>) -> Option<String> {
+ last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let offset_value = self.effective_offset(last_offset).unwrap_or_default();
+ let offset = format_offset_value(&offset_value, kind);
+ let quoted_table = quote_qualified_identifier(table)?;
+ let limit = batch_size.to_string();
+ let now = Utc::now();
+ let now_unix = now.timestamp().to_string();
+ let now = now.to_rfc3339();
+ let replacements = [
+ ("$table", quoted_table.as_str()),
+ ("$offset", offset.as_str()),
+ ("$limit", limit.as_str()),
+ ("$now_unix", now_unix.as_str()),
+ ("$now", now.as_str()),
+ ];
+
+ Ok(substitute_query_tokens(query, &replacements))
+ }
+
+ /// The query is substituted as text and re-executed once per configured table on
+ /// every poll for the life of the process, so a shape that cannot terminate does
+ /// not fail, it repeats. What can be judged from the text alone is judged here,
+ /// at startup, rather than left to the first poll.
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let tracking_column = self.tracking_column();
+
+ match leading_statement_token(query) {
+ None => {
+ return Err(Error::InitError(
+ "custom_query has no statement: it is empty, or nothing but comments"
+ .to_string(),
+ ));
+ }
+ Some(token) if !READ_STATEMENT_STARTS.contains(&token.as_str()) => {
+ return Err(Error::InitError(format!(
+ "custom_query must be a read starting with SELECT, WITH or TABLE, found \
+ '{token}'"
+ )));
+ }
+ Some(_) => {}
+ }
+
+ // Collapsed so `ORDER\nBY` and `ORDER BY` read the same as `ORDER BY`; the
+ // check hard-fails, so a formatting difference must not decide it.
+ let normalized = query
+ .split_whitespace()
+ .collect::<Vec<_>>()
+ .join(" ")
+ .to_uppercase();
+
+ if query.contains("$offset") {
+ if !normalized.contains("ORDER BY") {
+ return Err(Error::InitError(format!(
+ "custom_query filters on $offset but has no ORDER BY, add \
+ `ORDER BY {tracking_column}`"
+ )));
+ }
+ } else if !self.config.delete_after_read.unwrap_or(false)
+ && self.config.processed_column.is_none()
+ {
+ return Err(Error::InitError(format!(
+ "custom_query must filter with $offset (`WHERE {tracking_column} > $offset`), or \
+ set delete_after_read or processed_column"
+ )));
+ }
+
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "custom_query uses $table but no tables are configured".to_string(),
+ ));
+ }
+ // Without $table the same SQL is executed once per table, so identical rows
+ // would be emitted repeatedly, each carrying a different table's metadata,
+ // message ID, and tracking offset. Table identity is not recoverable from a
+ // static query, so reject the combination instead of guessing one.
+ if !query.contains("$table") && self.config.tables.len() > 1 {
+ return Err(Error::InitError(format!(
+ "custom_query has no $table placeholder but {} tables are configured ({}), use \
+ $table in the query or configure a single table",
+ self.config.tables.len(),
+ self.config.tables.join(", ")
+ )));
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = self.effective_offset(last_offset) {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(&offset, kind)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn apply_pending_operations(
+ &self,
+ operations: Vec<PendingOperation>,
+ deadline: tokio::time::Instant,
+ ) -> Result<Vec<PendingOperation>, Error> {
+ let mut failed_operations = Vec::new();
+ let mut operations = operations.into_iter();
+ while let Some(operation) = operations.next() {
+ if tokio::time::Instant::now() >= deadline {
+ failed_operations.push(operation);
+ failed_operations.extend(operations);
+ break;
+ }
+ let result = tokio::time::timeout_at(deadline, async {
+ let pool = self.get_pool()?;
+ self.mark_or_delete_processed_rows(pool, &operation).await
+ })
+ .await;
+ match result {
+ Ok(Ok(())) => {
+ self.consecutive_cleanup_failures
+ .store(0, Ordering::Relaxed);
+ }
+ Ok(Err(error)) => {
+ self.record_cleanup_failure(&operation, &error)?;
+ failed_operations.push(operation);
+ }
+ Err(_) => {
+ self.record_cleanup_failure(
+ &operation,
+ "cleanup exceeded the 10s budget for one Ack",
+ )?;
+ failed_operations.push(operation);
+ }
+ }
+ }
+ Ok(failed_operations)
+ }
+
+ fn record_cleanup_failure(
+ &self,
+ operation: &PendingOperation,
+ error: impl std::fmt::Display,
+ ) -> Result<(), Error> {
+ let consecutive_failures = self
+ .consecutive_cleanup_failures
+ .fetch_add(1, Ordering::Relaxed)
+ .saturating_add(1);
+ error!(
+ "Failed to mark or delete {} delivered row(s) in table '{}' for {CONNECTOR_NAME} \
+ connector ID: {}, retrying on the next poll ({consecutive_failures} consecutive \
+ failures): {error}",
+ operation.ids.len(),
+ operation.table,
+ self.id
+ );
+ if consecutive_failures >= MAX_CONSECUTIVE_CLEANUP_FAILURES {
+ return Err(Error::Connection(format!(
+ "stopping {CONNECTOR_NAME} connector ID {} after {consecutive_failures} \
+ consecutive mark/delete failures; the cleanup stays in the saved state and \
+ runs again on restart",
+ self.id
+ )));
+ }
+ Ok(())
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ operation: &PendingOperation,
+ ) -> Result<(), Error> {
+ let PendingOperation { table, ids, target } = operation;
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(&target.key_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = match &target.action {
+ CleanupAction::Delete => {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
+ }
+ CleanupAction::MarkProcessed { column } => {
+ let quoted_processed = quote_identifier(column)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({placeholders})"
+ )
+ }
+ };
+
+ // The query and its binds are rebuilt on every attempt because with_retry
+ // calls the closure once per try and each `execute` consumes the query.
+ with_retry(
+ || {
+ let mut prepared = sqlx::query(sqlx::AssertSqlSafe(query.as_str()));
+ for id in ids {
+ prepared = match id {
+ PkValue::Text(text) => prepared.bind(text),
+ PkValue::Bytes(bytes) => prepared.bind(bytes),
+ };
+ }
+ prepared.execute(pool)
+ },
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<PolledBatch, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.tracking_column();
+ let pk_column = self.primary_key_column();
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ //
+ // An ordering violation is handled one step further. Retrying it would
+ // re-run the same query, re-decode the same rows, and hit the same failure
+ // forever, so the table is latched off instead of being retried each cycle.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ let poll_number = self.polls.fetch_add(1, Ordering::Relaxed) + 1;
+ let poisoned = { self.poisoned_tables.lock().await.clone() };
+ let mut newly_poisoned: Vec<String> = Vec::new();
+
+ for table in &self.config.tables {
+ if poisoned.contains(table) {
+ continue;
+ }
+
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ if !batch.messages.is_empty() {
+ self.cursor_regressions.lock().await.remove(table);
+ }
+ // A full batch means the LIMIT was hit, so more rows are waiting.
+ any_batch_full |= batch.messages.len() as u32 >= batch_size;
+ batches.push(batch);
+ }
+ Err(FetchError::Ordering(reason) | FetchError::MissingTrackingColumn(reason)) => {
+ error!(
+ "Table '{table}' at offset {}: {reason}. Table disabled until restart.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ newly_poisoned.push(table.clone());
+ }
+ Err(FetchError::CursorRegression(reason)) => {
+ // Cleared only by a batch that carried rows, so the run counts polls
+ // since the table last published anything. An idle poll produces as
+ // little as a rejected one does and must not read as recovery.
+ let consecutive = {
+ let mut regressions = self.cursor_regressions.lock().await;
+ let consecutive = regressions.entry(table.clone()).or_insert(0);
+ *consecutive += 1;
+ *consecutive
+ };
+ let offset = last_offset.as_deref().unwrap_or("<start>");
+ if consecutive >= CURSOR_REGRESSION_ERROR_THRESHOLD {
+ error!(
+ "Table '{table}' at offset {offset}: {reason}. Skipping this poll \
+ ({consecutive} consecutive polls)."
+ );
+ } else {
+ warn!("Table '{table}' at offset {offset}: {reason}. Skipping this poll.");
+ }
+ }
+ Err(FetchError::StaleOffsetKind(reason)) => {
+ warn!(
+ "Table '{table}' at offset {}: {reason}. Retrying on the next poll.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::UnusableTrackingValue(reason)) => {
+ error!(
+ "Table '{table}' at offset {}: {reason}. Retrying each poll until the row \
+ is fixed.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::Other(error)) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ if !newly_poisoned.is_empty() {
+ let mut poisoned_tables = self.poisoned_tables.lock().await;
+ poisoned_tables.extend(newly_poisoned);
+ if self
+ .config
+ .tables
+ .iter()
+ .all(|table| poisoned_tables.contains(table))
+ {
+ error!(
+ "All configured tables are disabled because their batches cannot yield a \
+ cursor; {CONNECTOR_NAME} connector with ID: {} is idle until restarted",
+ self.id
+ );
+ }
+ }
+
+ // From the next poll on a disabled table is skipped before anything is logged,
+ // so the one line written when it was disabled can scroll out of the operator's
+ // log window and leave a dark connector looking healthy.
+ if poll_number.is_multiple_of(DISABLED_TABLES_LOG_INTERVAL) {
+ let poisoned_tables = self.poisoned_tables.lock().await;
+ if !poisoned_tables.is_empty() {
+ let mut disabled: Vec<&str> = poisoned_tables.iter().map(String::as_str).collect();
+ disabled.sort_unstable();
+ error!(
+ "{} of {} configured table(s) are still disabled and are not being polled by \
+ {CONNECTOR_NAME} connector with ID: {}, and will stay that way until it is \
+ restarted: {}",
+ disabled.len(),
+ self.config.tables.len(),
+ self.id,
+ disabled.join(", ")
+ );
+ }
+ }
+
+ // Phase 2: stage each table's batch. Nothing touches MySQL or the live state
+ // here: the offsets and the mark/delete are applied in `on_batch_result` once
+ // the runtime has sent the batch and saved this candidate state, and a Nack
+ // throws the candidate away so the same rows are read again.
+ let cleanup_target = self.cleanup_target();
+ let mut candidate_state = self.state.lock().await.clone();
+ let mut operations = Vec::new();
+ let mut messages = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if let Some(target) = &cleanup_target
+ && !batch.messages.is_empty()
+ {
+ if batch.processed_ids.is_empty() {
+ error!(
+ "Table '{}': mark/delete is configured but no primary keys were extracted \
+ from {} row(s), skipping this cycle so rows are not published without \
+ being marked or deleted (check that '{pk_column}' is projected and scalar)",
+ batch.table,
+ batch.messages.len()
+ );
+ continue;
+ }
+
+ match self
+ .cleanup_key_is_unique(pool, &batch.table, &target.key_column)
+ .await
+ {
+ Ok(true) => {}
+ Ok(false) => {
+ error!(
+ "{}. Table disabled until restart.",
+ not_unique_key_message(&batch.table, &target.key_column)
+ );
+ self.poisoned_tables.lock().await.insert(batch.table);
+ continue;
+ }
+ Err(error) => {
+ error!("Table '{}' skipped this cycle: {error}", batch.table);
+ continue;
+ }
+ }
+
+ operations.push(PendingOperation {
+ table: batch.table.clone(),
+ ids: std::mem::take(&mut batch.processed_ids),
+ target: target.clone(),
+ });
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ candidate_state.tracking_offsets.insert(batch.table, offset);
+ }
+ }
+
+ let pending = (total_processed > 0).then(|| {
+ candidate_state.processed_rows += total_processed;
+ candidate_state.pending_operations = operations;
+ PendingBatch {
+ state: candidate_state,
+ acknowledged: false,
+ retiring: false,
+ }
+ });
+
+ self.last_batch_full
+ .store(any_batch_full, Ordering::Relaxed);
+ Ok(PolledBatch { messages, pending })
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, FetchError> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query_kind = self.tracking_kind(table).await;
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.substitute_query_params(custom_query, table, last_offset, batch_size, query_kind)?
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size, query_kind)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ // Settled once for the whole batch rather than per row, since the column list
+ // is a property of the result set.
+ if let Some(first) = rows.first() {
+ ensure_tracking_column_projected(
+ first.columns().iter().map(|column| column.name()),
+ tracking_column,
+ )?;
+ }
+
+ let resume_from = if self.cursor_gates_query() {
+ self.effective_offset(last_offset)
+ } else {
+ None
+ };
+
+ // The result set describes what this query actually returned, so it outranks
+ // whatever `information_schema` said at startup and is the only answer
+ // available for a computed column.
+ let kind = match rows.first().and_then(|row| {
+ observed_tracking_kind(row, tracking_column).filter(|observed| *observed != query_kind)
+ }) {
+ Some(observed) => {
+ self.tracking_kinds
+ .lock()
+ .await
+ .insert(table.to_string(), observed);
+ // The query was already sent with the stale kind, so its offset literal
+ // may be quoted for the wrong type.
+ if let Some(offset) = &resume_from
+ && offset_literal_diverged(offset, query_kind, observed)
+ {
+ return Err(FetchError::StaleOffsetKind(format!(
+ "tracking_column '{tracking_column}' type differs from the one detected \
+ at startup (filtered with {}, expected {}), batch discarded",
+ format_offset_value(offset, query_kind),
+ format_offset_value(offset, observed)
+ )));
+ }
+ observed
+ }
+ None => query_kind,
+ };
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ // Every row of a result set shares its columns, so the published key names are
+ // derived once per batch instead of once per row.
+ let column_names: Vec<String> = rows
+ .first()
+ .map(|row| {
+ row.columns()
+ .iter()
+ .map(|column| {
+ if table_config.snake_case_columns {
+ to_snake_case(column.name())
+ } else {
+ column.name().to_string()
+ }
+ })
+ .collect()
+ })
+ .unwrap_or_default();
+ let mut ordering = OrderingGuard::new(tracking_column, kind, resume_from);
+ for row in rows {
+ let processed = self
+ .process_row(&row, &table_config, &column_names)
+ .map_err(|e| {
+ error!(
+ "Failed to decode row in table '{table}' (pk {}): {e}",
+ pk_for_log(&row, table_config.pk_column)
+ );
+ e
+ })?;
+
+ let Some(value) = processed.tracking_value else {
+ return Err(unusable_tracking_value(
+ tracking_column,
+ &pk_for_log(&row, table_config.pk_column),
+ ));
+ };
+ ordering
+ .accept(value)
+ .map_err(|violation| match violation {
+ OrderingViolation::WithinBatch(reason) => FetchError::Ordering(reason),
+ OrderingViolation::BelowCursor(reason) => FetchError::CursorRegression(reason),
+ })?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+
+ batch.messages.push(processed.message);
+ }
+ batch.max_offset = ordering.into_cursor();
+
+ Ok(batch)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ column_names: &[String],
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<PkValue> = None;
+ let mut tracking_value: Option<String> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
+ let mut extracted_payload: Option<Vec<u8>> = None;
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ if name.eq_ignore_ascii_case(config.payload_col) {
+ extracted_payload =
+ Some(self.extract_payload_column(row, i, config.payload_format)?);
+ }
+ let is_tracking = name.eq_ignore_ascii_case(config.tracking_column);
+ let is_pk = name.eq_ignore_ascii_case(config.pk_column);
+ if is_tracking {
+ let value = extract_column_value(row, i)?;
+ tracking_value = column_value_as_string(row, i, &value)?;
+ if is_pk {
+ row_pk = pk_value_from_decoded(row, i, &value)?;
+ }
+ } else if is_pk {
+ row_pk = extract_pk_value(row, i)?;
+ }
+ }
+
+ // Every extract_payload_column arm yields Ok for a NULL column, so None
+ // here means the column was absent. Falling back to the whole-row JSON
+ // would publish bytes contradicting the schema derived from payload_format.
+ let payload = extracted_payload.ok_or_else(|| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': payload_column '{}' not present in the result set",
+ config.table, config.payload_col
+ ))
+ })?;
+
+ return Ok(build_processed_row(
+ config.table,
+ payload,
+ tracking_value,
+ row_pk,
+ ));
+ }
+
+ let mut data = serde_json::Map::new();
+ for ((i, column), column_name) in row.columns().iter().enumerate().zip(column_names) {
+ let name = column.name();
+ let value = extract_column_value(row, i)?;
+ if name.eq_ignore_ascii_case(config.tracking_column) {
+ tracking_value = column_value_as_string(row, i, &value)?;
+ }
+ if name.eq_ignore_ascii_case(config.pk_column) {
+ row_pk = pk_value_from_decoded(row, i, &value)?;
+ }
+ data.insert(column_name.clone(), value);
+ }
+
+ let payload = if config.include_metadata {
+ let record = DatabaseRecord {
+ table_name: config.table.to_string(),
+ operation_type: "SELECT".to_string(),
+ timestamp: Utc::now(),
+ data: serde_json::Value::Object(data),
+ old_data: None,
+ };
+ simd_json::to_vec(&record).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': failed to serialize row to JSON: {e}",
+ config.table
+ ))
+ })?
+ } else {
+ simd_json::to_vec(&data).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': failed to serialize row to JSON: {e}",
+ config.table
+ ))
+ })?
+ };
+
+ Ok(build_processed_row(
+ config.table,
+ payload,
+ tracking_value,
+ row_pk,
+ ))
+ }
+}
+
+/// Type-faithful primary-key extraction. Binary columns are taken as raw bytes
+/// rather than routed through the base64 path in `extract_column_value`, so the
+/// value can be bound back into `WHERE pk IN (...)` and match the row. Everything
+/// else takes the string form MySQL parses back to the same value. Returns None
+/// only for a NULL pk, which a real primary key cannot be.
+fn extract_pk_value(row: &MySqlRow, column_index: usize) -> Result<Option<PkValue>, Error> {
+ if is_binary_column(row, column_index) {
+ return binary_pk_value(row, column_index);
+ }
+ let value = extract_column_value(row, column_index)?;
+ pk_value_from_decoded(row, column_index, &value)
+}
+
+/// Like `extract_pk_value` for a column the caller has already decoded, so the
+/// common key column is not decoded a second time per row.
+fn pk_value_from_decoded(
+ row: &MySqlRow,
+ column_index: usize,
+ value: &serde_json::Value,
+) -> Result<Option<PkValue>, Error> {
+ if is_binary_column(row, column_index) {
+ return binary_pk_value(row, column_index);
+ }
+ Ok(column_value_as_string(row, column_index, value)?.map(PkValue::Text))
+}
+
+fn is_binary_column(row: &MySqlRow, column_index: usize) -> bool {
+ matches!(
+ row.columns()[column_index].type_info().name(),
+ "BINARY" | "VARBINARY" | "TINYBLOB" | "BLOB" | "MEDIUMBLOB" | "LONGBLOB"
+ )
+}
+
+fn binary_pk_value(row: &MySqlRow, column_index: usize) -> Result<Option<PkValue>, Error> {
+ let column = &row.columns()[column_index];
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "primary key column '{}' (MySQL type '{}'): {e}",
+ column.name(),
+ column.type_info().name()
+ ))
+ })?;
+ Ok(bytes.map(PkValue::Bytes))
+}
+
+fn extract_column_value(row: &MySqlRow, column_index: usize) -> Result<serde_json::Value, Error> {
+ let column = &row.columns()[column_index];
+ let type_name = column.type_info().name();
+
+ let to_err = |e: sqlx::Error| {
+ Error::InvalidRecordValue(format!(
+ "column '{}' (MySQL type '{type_name}'): {e}",
+ column.name()
+ ))
+ };
+
+ match type_name {
+ "BOOLEAN" => {
+ let value: Option<bool> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::Bool)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT" => {
+ let value: Option<i8> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT" => {
+ let value: Option<i16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT" | "INT" => {
+ let value: Option<i32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT" => {
+ let value: Option<i64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT UNSIGNED" => {
+ let value: Option<u8> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT UNSIGNED" => {
+ let value: Option<u16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT UNSIGNED" | "INT UNSIGNED" => {
+ let value: Option<u32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT UNSIGNED" => {
+ let value: Option<u64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "FLOAT" => {
+ let value: Option<f32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as f64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DOUBLE" => {
+ let value: Option<f64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DECIMAL" => {
+ let value: Option<rust_decimal::Decimal> = row.try_get(column_index).map_err(to_err)?;
Review Comment:
critical: `Decimal` rounds values beyond 28 fractional digits, which changes payloads and can advance the cursor past unread rows. decode `DECIMAL` with `BigDecimal` to preserve the exact values.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,4071 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source,
+ source::SourceBatchResult, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+/// Upper bound on applying a batch's staged mark/delete after its Ack. The SDK waits
+/// `BATCH_RESULT_TIMEOUT` (30s) for the result, so this has to finish well inside it.
+const ACK_BATCH_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// MySQL rejects a prepared statement with more placeholders than this (error 1390), and
+/// the mark/delete binds one per row of the batch.
+const MAX_CLEANUP_BATCH_SIZE: u32 = 65_535;
+
+/// How many mark/delete statements in a row may fail before the connector stops. The
+/// operations stay in the persisted state and run again on restart.
+const MAX_CONSECUTIVE_CLEANUP_FAILURES: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ pending_batch: Mutex<Option<PendingBatch>>,
+ consecutive_cleanup_failures: AtomicU32,
+ verified_cleanup_keys: Mutex<HashSet<String>>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some("it has too few distinct values to be unique per row");
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some("binary values are compared as base64 text, which does not keep byte order");
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some("MySQL orders these by declaration index, not by label");
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL \
+ or not an ordered scalar (tinyint(1), JSON)"
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "row '{value}' is below the stored offset '{previous}' for tracking_column \
+ '{column}'. Filter the query with `{column} > $offset`"
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. Order the query ascending by '{column}'"
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+ /// Mark/delete work for a batch the runtime has already sent. It is persisted with
+ /// the batch's offsets so a crash between the send and the cleanup reruns it.
+ #[serde(default)]
+ pending_operations: Vec<PendingOperation>,
+}
+
+/// The rows of one table to mark or delete once their batch is acknowledged.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct PendingOperation {
+ table: String,
+ ids: Vec<PkValue>,
+ target: CleanupTarget,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct CleanupTarget {
+ key_column: String,
+ action: CleanupAction,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum CleanupAction {
+ Delete,
+ MarkProcessed { column: String },
+}
+
+/// The candidate state of the batch most recently handed to the runtime. Live state
+/// only takes it over on Ack, so a Nack leaves offsets and rows where they were.
+#[derive(Debug, Clone)]
+struct PendingBatch {
+ state: State,
+ /// The runtime has delivered the batch but its cleanup has not finished yet.
+ acknowledged: bool,
+ /// Cleanup finished and the next poll must persist `state` without the operations,
+ /// so a restart does not run them again.
+ retiring: bool,
+}
+
+struct PolledBatch {
+ messages: Vec<ProducedMessage>,
+ pending: Option<PendingBatch>,
+}
+
+enum CleanupKeyCheck {
+ Unique,
+ NotUnique,
+ TableMissing,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty, list the table even when custom_query hardcodes it"
+ .to_string(),
+ ));
+ }
+
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ if let Some(batch_size) = self.config.batch_size
+ && batch_size > MAX_CLEANUP_BATCH_SIZE
+ && self.cleanup_target().is_some()
+ {
+ return Err(Error::InitError(format!(
+ "batch_size {batch_size} exceeds {MAX_CLEANUP_BATCH_SIZE} with delete_after_read or \
+ processed_column set, MySQL allows at most {MAX_CLEANUP_BATCH_SIZE} placeholders \
+ in the statement that marks or deletes a batch"
+ )));
+ }
+
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "name the column custom_query orders by and filters with $offset"
+ } else {
+ "use a unique, monotonically increasing column"
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set, {guidance}"
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.validate_pending_cleanup()?;
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+ self.validate_cleanup_keys().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ if let Some(state) = self.retiring_state().await {
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: Some(self.serialize_pending_state(&state)?),
+ });
+ }
+
+ // Skip the pacing delay while draining a backlog, but never while a cleanup
+ // retry is pending, or a failing mark/delete would spin without a pause.
+ let cleanup_pending = self.pending_batch.lock().await.is_some();
+ if cleanup_pending || !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ {
+ let pending = self.pending_batch.lock().await;
+ if let Some(pending) = pending.as_ref() {
+ if !pending.acknowledged {
+ error!(
+ "{CONNECTOR_NAME} connector ID: {} was polled while a batch was still in flight",
+ self.id
+ );
+ return Err(Error::InvalidState);
+ }
+ // An empty batch whose Ack retries the cleanup the last one left behind.
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: None,
+ });
+ }
+ }
+
+ let polled = self.poll_tables().await?;
+
+ let processed_rows = match &polled.pending {
+ Some(pending) => pending.state.processed_rows,
+ None => self.state.lock().await.processed_rows,
+ };
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ }
+
+ // Idle cycles stage nothing, so return None to let the runtime skip the fsync.
+ let persisted_state = polled
+ .pending
+ .as_ref()
+ .map(|pending| self.serialize_pending_state(&pending.state))
+ .transpose()?;
+ *self.pending_batch.lock().await = polled.pending;
+
+ Ok(ProducedMessages {
+ schema,
+ messages: polled.messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> {
+ if result == SourceBatchResult::Nack {
+ // Dropping the candidate leaves the live offsets where they were, so the
+ // next poll reads the same rows again. Delivered batches keep their cleanup.
+ let mut pending = self.pending_batch.lock().await;
+ if pending
+ .as_ref()
+ .is_some_and(|pending| !pending.acknowledged && !pending.retiring)
+ {
+ pending.take();
+ }
+ return Ok(());
+ }
+
+ let Some(PendingBatch { mut state, .. }) = self.pending_batch.lock().await.clone() else {
+ return Ok(());
+ };
+
+ let operations = std::mem::take(&mut state.pending_operations);
+ let had_operations = !operations.is_empty();
+ let deadline = tokio::time::Instant::now() + ACK_BATCH_TIMEOUT;
+ let failed_operations = self.apply_pending_operations(operations, deadline).await?;
+
+ if failed_operations.is_empty() {
+ *self.state.lock().await = state.clone();
+ let mut pending = self.pending_batch.lock().await;
+ if had_operations {
+ *pending = Some(PendingBatch {
+ state,
+ acknowledged: false,
+ retiring: true,
+ });
+ } else {
+ pending.take();
+ }
+ } else {
+ state.pending_operations = failed_operations;
+ *self.pending_batch.lock().await = Some(PendingBatch {
+ state,
+ acknowledged: true,
+ retiring: false,
+ });
+ }
+
+ Ok(())
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ let state = restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ pending_operations: Vec::new(),
+ });
+ // The runtime only persists a batch's operations after sending it, so restored
+ // operations belong to a delivered batch and only the cleanup is left to do.
+ let pending_batch = (!state.pending_operations.is_empty()).then(|| PendingBatch {
+ state: state.clone(),
+ acknowledged: true,
+ retiring: false,
+ });
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(state),
+ pending_batch: Mutex::new(pending_batch),
+ consecutive_cleanup_failures: AtomicU32::new(0),
+ verified_cleanup_keys: Mutex::new(HashSet::new()),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ poisoned_tables: Mutex::new(HashSet::new()),
+ tracking_kinds: Mutex::new(HashMap::new()),
+ polls: AtomicU64::new(0),
+ cursor_regressions: Mutex::new(HashMap::new()),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ // `format_offset_value` escapes backslashes, which under NO_BACKSLASH_ESCAPES
+ // would reach MySQL as literal characters and filter on the wrong cursor.
+ .after_connect(|connection, _| {
+ Box::pin(async move {
+ sqlx::query(
+ "SET SESSION sql_mode = REPLACE(@@SESSION.sql_mode, 'NO_BACKSLASH_ESCAPES', '')",
+ )
+ .execute(&mut *connection)
+ .await?;
+ Ok(())
+ })
+ })
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn tracking_column(&self) -> &str {
+ self.config.tracking_column.as_deref().unwrap_or("id")
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn resolve_tracking_columns(&self) -> Result<HashMap<String, OffsetKind>, Error> {
+ // Whether the connector owns the query, and can therefore hold the schema to
+ // what that query will do with it.
+ let strict = self.config.custom_query.is_none();
+ let pool = self.get_pool()?;
+ let tracking_column = self.tracking_column();
+ let mut kinds = HashMap::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ }
+
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ }
+
+ warn!(
+ "tracking_column '{tracking_column}' is not a column of table '{table}'; \
+ assuming custom_query projects it. Its comparison order is resolved from \
+ the first result set instead."
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ // The column's type survives a plain projection, so it is worth rejecting
+ // whatever the query looks like. A custom_query aliasing an expression onto
+ // a name the table also uses is the one false positive, hence the hint.
+ if let Some(reason) = unusable_tracking_type(&data_type) {
+ let aliasing_hint = if self.config.custom_query.is_some() {
+ " If custom_query aliases a different expression to this name, rename the \
+ alias so it does not collide with the column."
+ } else {
+ ""
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': {reason}. Use an integer, timestamp, or string \
+ column.{aliasing_hint}"
+ )));
+ }
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; a row \
+ whose value is NULL cannot produce a cursor and will fail that table's batch. \
+ Have custom_query exclude those rows, or declare the column NOT NULL."
+ );
+ }
+
+ if is_collation_ordered(&data_type) {
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': values must be unique and increase in its collation order \
+ (zero-pad numbers, prefer a _bin collation)"
+ );
+ }
+
+ kinds.insert(table.clone(), offset_kind_for_data_type(&data_type));
+ }
+ Ok(kinds)
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn serialize_pending_state(&self, state: &State) -> Result<ConnectorState, Error> {
+ self.serialize_state(state).ok_or_else(|| {
+ Error::Serialization(format!("failed to serialize {CONNECTOR_NAME} state"))
+ })
+ }
+
+ async fn retiring_state(&self) -> Option<State> {
+ self.pending_batch
+ .lock()
+ .await
+ .as_ref()
+ .filter(|pending| pending.retiring)
+ .map(|pending| pending.state.clone())
+ }
+
+ fn primary_key_column(&self) -> &str {
+ self.config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(self.tracking_column())
+ }
+
+ fn cleanup_target(&self) -> Option<CleanupTarget> {
+ let action = if self.config.delete_after_read.unwrap_or(false) {
+ CleanupAction::Delete
+ } else if let Some(column) = &self.config.processed_column {
+ CleanupAction::MarkProcessed {
+ column: column.clone(),
+ }
+ } else {
+ return None;
+ };
+ Some(CleanupTarget {
+ key_column: self.primary_key_column().to_string(),
+ action,
+ })
+ }
+
+ /// Restored operations were staged under the config of the run that wrote them.
+ /// Running them under a different key or action would touch rows nobody read.
+ fn validate_pending_cleanup(&mut self) -> Result<(), Error> {
+ let configured = self.cleanup_target();
+ for operation in &self.state.get_mut().pending_operations {
+ if configured.as_ref() != Some(&operation.target) {
+ return Err(Error::InitError(format!(
+ "the cleanup configuration changed while cleanup of {} delivered row(s) in \
+ table '{}' is pending; restore primary_key_column '{}' and action {:?}",
+ operation.ids.len(),
+ operation.table,
+ operation.target.key_column,
+ operation.target.action
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ /// `WHERE key IN (...)` on a key that repeats marks or deletes every row sharing a
+ /// value, including ones that were never read. Tables that don't exist yet are
+ /// checked by the first poll that stages cleanup for them.
+ async fn validate_cleanup_keys(&mut self) -> Result<(), Error> {
+ let Some(target) = self.cleanup_target() else {
+ return Ok(());
+ };
+ let pool = self.get_pool()?;
+ let mut verified = HashSet::with_capacity(self.config.tables.len());
+ for table in &self.config.tables {
+ let check = cleanup_key_check(pool, table, &target.key_column)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to check primary_key_column '{}' on table '{table}': {e}",
+ target.key_column
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ verified.insert(table.clone());
+ }
+ CleanupKeyCheck::NotUnique => {
+ return Err(Error::InitError(not_unique_key_message(
+ table,
+ &target.key_column,
+ )));
+ }
+ CleanupKeyCheck::TableMissing => {}
+ }
+ }
+ *self.verified_cleanup_keys.get_mut() = verified;
+ Ok(())
+ }
+
+ async fn cleanup_key_is_unique(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ key_column: &str,
+ ) -> Result<bool, Error> {
+ if self.verified_cleanup_keys.lock().await.contains(table) {
+ return Ok(true);
+ }
+ let check = cleanup_key_check(pool, table, key_column)
+ .await
+ .map_err(|e| {
+ Error::Connection(format!(
+ "failed to check primary_key_column '{key_column}' on table '{table}': {e}"
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ self.verified_cleanup_keys
+ .lock()
+ .await
+ .insert(table.to_string());
+ Ok(true)
+ }
+ CleanupKeyCheck::NotUnique => Ok(false),
+ CleanupKeyCheck::TableMissing => Err(Error::Connection(format!(
+ "table '{table}' disappeared before its primary_key_column could be checked"
+ ))),
+ }
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ /// Best current knowledge of how a table's tracking values compare.
+ async fn tracking_kind(&self, table: &str) -> OffsetKind {
+ self.tracking_kinds
+ .lock()
+ .await
+ .get(table)
+ .copied()
+ .unwrap_or(OffsetKind::Unknown)
+ }
+
+ /// Whether the stored cursor is what limits the rows a poll can see. The built
+ /// polling query always filters on it; a `custom_query` does so only if it
+ /// interpolates `$offset`.
+ fn cursor_gates_query(&self) -> bool {
+ match &self.config.custom_query {
+ Some(query) => query.contains("$offset"),
+ None => true,
+ }
+ }
+
+ /// The offset a poll actually resumes from: the stored cursor, or the configured
+ /// starting point before anything has been stored.
+ fn effective_offset(&self, last_offset: &Option<String>) -> Option<String> {
+ last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let offset_value = self.effective_offset(last_offset).unwrap_or_default();
+ let offset = format_offset_value(&offset_value, kind);
+ let quoted_table = quote_qualified_identifier(table)?;
+ let limit = batch_size.to_string();
+ let now = Utc::now();
+ let now_unix = now.timestamp().to_string();
+ let now = now.to_rfc3339();
+ let replacements = [
+ ("$table", quoted_table.as_str()),
+ ("$offset", offset.as_str()),
+ ("$limit", limit.as_str()),
+ ("$now_unix", now_unix.as_str()),
+ ("$now", now.as_str()),
+ ];
+
+ Ok(substitute_query_tokens(query, &replacements))
+ }
+
+ /// The query is substituted as text and re-executed once per configured table on
+ /// every poll for the life of the process, so a shape that cannot terminate does
+ /// not fail, it repeats. What can be judged from the text alone is judged here,
+ /// at startup, rather than left to the first poll.
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let tracking_column = self.tracking_column();
+
+ match leading_statement_token(query) {
+ None => {
+ return Err(Error::InitError(
+ "custom_query has no statement: it is empty, or nothing but comments"
+ .to_string(),
+ ));
+ }
+ Some(token) if !READ_STATEMENT_STARTS.contains(&token.as_str()) => {
+ return Err(Error::InitError(format!(
+ "custom_query must be a read starting with SELECT, WITH or TABLE, found \
+ '{token}'"
+ )));
+ }
+ Some(_) => {}
+ }
+
+ // Collapsed so `ORDER\nBY` and `ORDER BY` read the same as `ORDER BY`; the
+ // check hard-fails, so a formatting difference must not decide it.
+ let normalized = query
+ .split_whitespace()
+ .collect::<Vec<_>>()
+ .join(" ")
+ .to_uppercase();
+
+ if query.contains("$offset") {
+ if !normalized.contains("ORDER BY") {
+ return Err(Error::InitError(format!(
+ "custom_query filters on $offset but has no ORDER BY, add \
+ `ORDER BY {tracking_column}`"
+ )));
+ }
+ } else if !self.config.delete_after_read.unwrap_or(false)
+ && self.config.processed_column.is_none()
+ {
+ return Err(Error::InitError(format!(
+ "custom_query must filter with $offset (`WHERE {tracking_column} > $offset`), or \
+ set delete_after_read or processed_column"
+ )));
+ }
+
+ if query.contains("$table") && self.config.tables.is_empty() {
Review Comment:
simplification: `validate_custom_query()` repeats the empty-table rejection that `open()` already enforces. remove the unreachable branch and the test that calls it directly.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,4071 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source,
+ source::SourceBatchResult, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+/// Upper bound on applying a batch's staged mark/delete after its Ack. The SDK waits
+/// `BATCH_RESULT_TIMEOUT` (30s) for the result, so this has to finish well inside it.
+const ACK_BATCH_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// MySQL rejects a prepared statement with more placeholders than this (error 1390), and
+/// the mark/delete binds one per row of the batch.
+const MAX_CLEANUP_BATCH_SIZE: u32 = 65_535;
+
+/// How many mark/delete statements in a row may fail before the connector stops. The
+/// operations stay in the persisted state and run again on restart.
+const MAX_CONSECUTIVE_CLEANUP_FAILURES: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ pending_batch: Mutex<Option<PendingBatch>>,
+ consecutive_cleanup_failures: AtomicU32,
+ verified_cleanup_keys: Mutex<HashSet<String>>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some("it has too few distinct values to be unique per row");
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some("binary values are compared as base64 text, which does not keep byte order");
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some("MySQL orders these by declaration index, not by label");
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL \
+ or not an ordered scalar (tinyint(1), JSON)"
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "row '{value}' is below the stored offset '{previous}' for tracking_column \
+ '{column}'. Filter the query with `{column} > $offset`"
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. Order the query ascending by '{column}'"
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+ /// Mark/delete work for a batch the runtime has already sent. It is persisted with
+ /// the batch's offsets so a crash between the send and the cleanup reruns it.
+ #[serde(default)]
+ pending_operations: Vec<PendingOperation>,
+}
+
+/// The rows of one table to mark or delete once their batch is acknowledged.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct PendingOperation {
+ table: String,
+ ids: Vec<PkValue>,
+ target: CleanupTarget,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct CleanupTarget {
+ key_column: String,
+ action: CleanupAction,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum CleanupAction {
+ Delete,
+ MarkProcessed { column: String },
+}
+
+/// The candidate state of the batch most recently handed to the runtime. Live state
+/// only takes it over on Ack, so a Nack leaves offsets and rows where they were.
+#[derive(Debug, Clone)]
+struct PendingBatch {
+ state: State,
+ /// The runtime has delivered the batch but its cleanup has not finished yet.
+ acknowledged: bool,
+ /// Cleanup finished and the next poll must persist `state` without the operations,
+ /// so a restart does not run them again.
+ retiring: bool,
+}
+
+struct PolledBatch {
+ messages: Vec<ProducedMessage>,
+ pending: Option<PendingBatch>,
+}
+
+enum CleanupKeyCheck {
+ Unique,
+ NotUnique,
+ TableMissing,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty, list the table even when custom_query hardcodes it"
+ .to_string(),
+ ));
+ }
+
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ if let Some(batch_size) = self.config.batch_size
+ && batch_size > MAX_CLEANUP_BATCH_SIZE
+ && self.cleanup_target().is_some()
+ {
+ return Err(Error::InitError(format!(
+ "batch_size {batch_size} exceeds {MAX_CLEANUP_BATCH_SIZE} with delete_after_read or \
+ processed_column set, MySQL allows at most {MAX_CLEANUP_BATCH_SIZE} placeholders \
+ in the statement that marks or deletes a batch"
+ )));
+ }
+
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "name the column custom_query orders by and filters with $offset"
+ } else {
+ "use a unique, monotonically increasing column"
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set, {guidance}"
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.validate_pending_cleanup()?;
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+ self.validate_cleanup_keys().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ if let Some(state) = self.retiring_state().await {
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: Some(self.serialize_pending_state(&state)?),
+ });
+ }
+
+ // Skip the pacing delay while draining a backlog, but never while a cleanup
+ // retry is pending, or a failing mark/delete would spin without a pause.
+ let cleanup_pending = self.pending_batch.lock().await.is_some();
+ if cleanup_pending || !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ {
+ let pending = self.pending_batch.lock().await;
+ if let Some(pending) = pending.as_ref() {
+ if !pending.acknowledged {
+ error!(
+ "{CONNECTOR_NAME} connector ID: {} was polled while a batch was still in flight",
+ self.id
+ );
+ return Err(Error::InvalidState);
+ }
+ // An empty batch whose Ack retries the cleanup the last one left behind.
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: None,
+ });
+ }
+ }
+
+ let polled = self.poll_tables().await?;
+
+ let processed_rows = match &polled.pending {
+ Some(pending) => pending.state.processed_rows,
+ None => self.state.lock().await.processed_rows,
+ };
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ }
+
+ // Idle cycles stage nothing, so return None to let the runtime skip the fsync.
+ let persisted_state = polled
+ .pending
+ .as_ref()
+ .map(|pending| self.serialize_pending_state(&pending.state))
+ .transpose()?;
+ *self.pending_batch.lock().await = polled.pending;
+
+ Ok(ProducedMessages {
+ schema,
+ messages: polled.messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> {
+ if result == SourceBatchResult::Nack {
+ // Dropping the candidate leaves the live offsets where they were, so the
+ // next poll reads the same rows again. Delivered batches keep their cleanup.
+ let mut pending = self.pending_batch.lock().await;
+ if pending
+ .as_ref()
+ .is_some_and(|pending| !pending.acknowledged && !pending.retiring)
+ {
+ pending.take();
+ }
+ return Ok(());
+ }
+
+ let Some(PendingBatch { mut state, .. }) = self.pending_batch.lock().await.clone() else {
+ return Ok(());
+ };
+
+ let operations = std::mem::take(&mut state.pending_operations);
+ let had_operations = !operations.is_empty();
+ let deadline = tokio::time::Instant::now() + ACK_BATCH_TIMEOUT;
+ let failed_operations = self.apply_pending_operations(operations, deadline).await?;
+
+ if failed_operations.is_empty() {
+ *self.state.lock().await = state.clone();
+ let mut pending = self.pending_batch.lock().await;
+ if had_operations {
+ *pending = Some(PendingBatch {
+ state,
+ acknowledged: false,
+ retiring: true,
+ });
+ } else {
+ pending.take();
+ }
+ } else {
+ state.pending_operations = failed_operations;
+ *self.pending_batch.lock().await = Some(PendingBatch {
+ state,
+ acknowledged: true,
+ retiring: false,
+ });
+ }
+
+ Ok(())
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ let state = restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ pending_operations: Vec::new(),
+ });
+ // The runtime only persists a batch's operations after sending it, so restored
+ // operations belong to a delivered batch and only the cleanup is left to do.
+ let pending_batch = (!state.pending_operations.is_empty()).then(|| PendingBatch {
+ state: state.clone(),
+ acknowledged: true,
+ retiring: false,
+ });
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(state),
+ pending_batch: Mutex::new(pending_batch),
+ consecutive_cleanup_failures: AtomicU32::new(0),
+ verified_cleanup_keys: Mutex::new(HashSet::new()),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ poisoned_tables: Mutex::new(HashSet::new()),
+ tracking_kinds: Mutex::new(HashMap::new()),
+ polls: AtomicU64::new(0),
+ cursor_regressions: Mutex::new(HashMap::new()),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ // `format_offset_value` escapes backslashes, which under NO_BACKSLASH_ESCAPES
+ // would reach MySQL as literal characters and filter on the wrong cursor.
+ .after_connect(|connection, _| {
+ Box::pin(async move {
+ sqlx::query(
+ "SET SESSION sql_mode = REPLACE(@@SESSION.sql_mode, 'NO_BACKSLASH_ESCAPES', '')",
+ )
+ .execute(&mut *connection)
+ .await?;
+ Ok(())
+ })
+ })
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn tracking_column(&self) -> &str {
+ self.config.tracking_column.as_deref().unwrap_or("id")
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn resolve_tracking_columns(&self) -> Result<HashMap<String, OffsetKind>, Error> {
+ // Whether the connector owns the query, and can therefore hold the schema to
+ // what that query will do with it.
+ let strict = self.config.custom_query.is_none();
+ let pool = self.get_pool()?;
+ let tracking_column = self.tracking_column();
+ let mut kinds = HashMap::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ }
+
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ }
+
+ warn!(
+ "tracking_column '{tracking_column}' is not a column of table '{table}'; \
+ assuming custom_query projects it. Its comparison order is resolved from \
+ the first result set instead."
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ // The column's type survives a plain projection, so it is worth rejecting
+ // whatever the query looks like. A custom_query aliasing an expression onto
+ // a name the table also uses is the one false positive, hence the hint.
+ if let Some(reason) = unusable_tracking_type(&data_type) {
+ let aliasing_hint = if self.config.custom_query.is_some() {
+ " If custom_query aliases a different expression to this name, rename the \
+ alias so it does not collide with the column."
+ } else {
+ ""
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': {reason}. Use an integer, timestamp, or string \
+ column.{aliasing_hint}"
+ )));
+ }
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; a row \
+ whose value is NULL cannot produce a cursor and will fail that table's batch. \
+ Have custom_query exclude those rows, or declare the column NOT NULL."
+ );
+ }
+
+ if is_collation_ordered(&data_type) {
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': values must be unique and increase in its collation order \
+ (zero-pad numbers, prefer a _bin collation)"
+ );
+ }
+
+ kinds.insert(table.clone(), offset_kind_for_data_type(&data_type));
+ }
+ Ok(kinds)
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn serialize_pending_state(&self, state: &State) -> Result<ConnectorState, Error> {
+ self.serialize_state(state).ok_or_else(|| {
+ Error::Serialization(format!("failed to serialize {CONNECTOR_NAME} state"))
+ })
+ }
+
+ async fn retiring_state(&self) -> Option<State> {
+ self.pending_batch
+ .lock()
+ .await
+ .as_ref()
+ .filter(|pending| pending.retiring)
+ .map(|pending| pending.state.clone())
+ }
+
+ fn primary_key_column(&self) -> &str {
+ self.config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(self.tracking_column())
+ }
+
+ fn cleanup_target(&self) -> Option<CleanupTarget> {
+ let action = if self.config.delete_after_read.unwrap_or(false) {
+ CleanupAction::Delete
+ } else if let Some(column) = &self.config.processed_column {
+ CleanupAction::MarkProcessed {
+ column: column.clone(),
+ }
+ } else {
+ return None;
+ };
+ Some(CleanupTarget {
+ key_column: self.primary_key_column().to_string(),
+ action,
+ })
+ }
+
+ /// Restored operations were staged under the config of the run that wrote them.
+ /// Running them under a different key or action would touch rows nobody read.
+ fn validate_pending_cleanup(&mut self) -> Result<(), Error> {
+ let configured = self.cleanup_target();
+ for operation in &self.state.get_mut().pending_operations {
+ if configured.as_ref() != Some(&operation.target) {
+ return Err(Error::InitError(format!(
+ "the cleanup configuration changed while cleanup of {} delivered row(s) in \
+ table '{}' is pending; restore primary_key_column '{}' and action {:?}",
+ operation.ids.len(),
+ operation.table,
+ operation.target.key_column,
+ operation.target.action
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ /// `WHERE key IN (...)` on a key that repeats marks or deletes every row sharing a
+ /// value, including ones that were never read. Tables that don't exist yet are
+ /// checked by the first poll that stages cleanup for them.
+ async fn validate_cleanup_keys(&mut self) -> Result<(), Error> {
+ let Some(target) = self.cleanup_target() else {
+ return Ok(());
+ };
+ let pool = self.get_pool()?;
+ let mut verified = HashSet::with_capacity(self.config.tables.len());
+ for table in &self.config.tables {
+ let check = cleanup_key_check(pool, table, &target.key_column)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to check primary_key_column '{}' on table '{table}': {e}",
+ target.key_column
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ verified.insert(table.clone());
+ }
+ CleanupKeyCheck::NotUnique => {
+ return Err(Error::InitError(not_unique_key_message(
+ table,
+ &target.key_column,
+ )));
+ }
+ CleanupKeyCheck::TableMissing => {}
+ }
+ }
+ *self.verified_cleanup_keys.get_mut() = verified;
+ Ok(())
+ }
+
+ async fn cleanup_key_is_unique(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ key_column: &str,
+ ) -> Result<bool, Error> {
+ if self.verified_cleanup_keys.lock().await.contains(table) {
+ return Ok(true);
+ }
+ let check = cleanup_key_check(pool, table, key_column)
+ .await
+ .map_err(|e| {
+ Error::Connection(format!(
+ "failed to check primary_key_column '{key_column}' on table '{table}': {e}"
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ self.verified_cleanup_keys
+ .lock()
+ .await
+ .insert(table.to_string());
+ Ok(true)
+ }
+ CleanupKeyCheck::NotUnique => Ok(false),
+ CleanupKeyCheck::TableMissing => Err(Error::Connection(format!(
+ "table '{table}' disappeared before its primary_key_column could be checked"
+ ))),
+ }
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ /// Best current knowledge of how a table's tracking values compare.
+ async fn tracking_kind(&self, table: &str) -> OffsetKind {
+ self.tracking_kinds
+ .lock()
+ .await
+ .get(table)
+ .copied()
+ .unwrap_or(OffsetKind::Unknown)
+ }
+
+ /// Whether the stored cursor is what limits the rows a poll can see. The built
+ /// polling query always filters on it; a `custom_query` does so only if it
+ /// interpolates `$offset`.
+ fn cursor_gates_query(&self) -> bool {
+ match &self.config.custom_query {
+ Some(query) => query.contains("$offset"),
+ None => true,
+ }
+ }
+
+ /// The offset a poll actually resumes from: the stored cursor, or the configured
+ /// starting point before anything has been stored.
+ fn effective_offset(&self, last_offset: &Option<String>) -> Option<String> {
+ last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let offset_value = self.effective_offset(last_offset).unwrap_or_default();
+ let offset = format_offset_value(&offset_value, kind);
+ let quoted_table = quote_qualified_identifier(table)?;
+ let limit = batch_size.to_string();
+ let now = Utc::now();
+ let now_unix = now.timestamp().to_string();
+ let now = now.to_rfc3339();
+ let replacements = [
+ ("$table", quoted_table.as_str()),
+ ("$offset", offset.as_str()),
+ ("$limit", limit.as_str()),
+ ("$now_unix", now_unix.as_str()),
+ ("$now", now.as_str()),
+ ];
+
+ Ok(substitute_query_tokens(query, &replacements))
+ }
+
+ /// The query is substituted as text and re-executed once per configured table on
+ /// every poll for the life of the process, so a shape that cannot terminate does
+ /// not fail, it repeats. What can be judged from the text alone is judged here,
+ /// at startup, rather than left to the first poll.
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let tracking_column = self.tracking_column();
+
+ match leading_statement_token(query) {
+ None => {
+ return Err(Error::InitError(
+ "custom_query has no statement: it is empty, or nothing but comments"
+ .to_string(),
+ ));
+ }
+ Some(token) if !READ_STATEMENT_STARTS.contains(&token.as_str()) => {
+ return Err(Error::InitError(format!(
+ "custom_query must be a read starting with SELECT, WITH or TABLE, found \
+ '{token}'"
+ )));
+ }
+ Some(_) => {}
+ }
+
+ // Collapsed so `ORDER\nBY` and `ORDER BY` read the same as `ORDER BY`; the
+ // check hard-fails, so a formatting difference must not decide it.
+ let normalized = query
+ .split_whitespace()
+ .collect::<Vec<_>>()
+ .join(" ")
+ .to_uppercase();
+
+ if query.contains("$offset") {
+ if !normalized.contains("ORDER BY") {
+ return Err(Error::InitError(format!(
+ "custom_query filters on $offset but has no ORDER BY, add \
+ `ORDER BY {tracking_column}`"
+ )));
+ }
+ } else if !self.config.delete_after_read.unwrap_or(false)
+ && self.config.processed_column.is_none()
+ {
+ return Err(Error::InitError(format!(
+ "custom_query must filter with $offset (`WHERE {tracking_column} > $offset`), or \
+ set delete_after_read or processed_column"
+ )));
+ }
+
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "custom_query uses $table but no tables are configured".to_string(),
+ ));
+ }
+ // Without $table the same SQL is executed once per table, so identical rows
+ // would be emitted repeatedly, each carrying a different table's metadata,
+ // message ID, and tracking offset. Table identity is not recoverable from a
+ // static query, so reject the combination instead of guessing one.
+ if !query.contains("$table") && self.config.tables.len() > 1 {
+ return Err(Error::InitError(format!(
+ "custom_query has no $table placeholder but {} tables are configured ({}), use \
+ $table in the query or configure a single table",
+ self.config.tables.len(),
+ self.config.tables.join(", ")
+ )));
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = self.effective_offset(last_offset) {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(&offset, kind)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn apply_pending_operations(
+ &self,
+ operations: Vec<PendingOperation>,
+ deadline: tokio::time::Instant,
+ ) -> Result<Vec<PendingOperation>, Error> {
+ let mut failed_operations = Vec::new();
+ let mut operations = operations.into_iter();
+ while let Some(operation) = operations.next() {
+ if tokio::time::Instant::now() >= deadline {
+ failed_operations.push(operation);
+ failed_operations.extend(operations);
+ break;
+ }
+ let result = tokio::time::timeout_at(deadline, async {
+ let pool = self.get_pool()?;
+ self.mark_or_delete_processed_rows(pool, &operation).await
+ })
+ .await;
+ match result {
+ Ok(Ok(())) => {
+ self.consecutive_cleanup_failures
+ .store(0, Ordering::Relaxed);
+ }
+ Ok(Err(error)) => {
+ self.record_cleanup_failure(&operation, &error)?;
+ failed_operations.push(operation);
+ }
+ Err(_) => {
+ self.record_cleanup_failure(
+ &operation,
+ "cleanup exceeded the 10s budget for one Ack",
+ )?;
+ failed_operations.push(operation);
+ }
+ }
+ }
+ Ok(failed_operations)
+ }
+
+ fn record_cleanup_failure(
+ &self,
+ operation: &PendingOperation,
+ error: impl std::fmt::Display,
+ ) -> Result<(), Error> {
+ let consecutive_failures = self
+ .consecutive_cleanup_failures
+ .fetch_add(1, Ordering::Relaxed)
+ .saturating_add(1);
+ error!(
+ "Failed to mark or delete {} delivered row(s) in table '{}' for {CONNECTOR_NAME} \
+ connector ID: {}, retrying on the next poll ({consecutive_failures} consecutive \
+ failures): {error}",
+ operation.ids.len(),
+ operation.table,
+ self.id
+ );
+ if consecutive_failures >= MAX_CONSECUTIVE_CLEANUP_FAILURES {
+ return Err(Error::Connection(format!(
+ "stopping {CONNECTOR_NAME} connector ID {} after {consecutive_failures} \
+ consecutive mark/delete failures; the cleanup stays in the saved state and \
+ runs again on restart",
+ self.id
+ )));
+ }
+ Ok(())
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ operation: &PendingOperation,
+ ) -> Result<(), Error> {
+ let PendingOperation { table, ids, target } = operation;
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(&target.key_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = match &target.action {
+ CleanupAction::Delete => {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
+ }
+ CleanupAction::MarkProcessed { column } => {
+ let quoted_processed = quote_identifier(column)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({placeholders})"
+ )
+ }
+ };
+
+ // The query and its binds are rebuilt on every attempt because with_retry
+ // calls the closure once per try and each `execute` consumes the query.
+ with_retry(
+ || {
+ let mut prepared = sqlx::query(sqlx::AssertSqlSafe(query.as_str()));
+ for id in ids {
+ prepared = match id {
+ PkValue::Text(text) => prepared.bind(text),
+ PkValue::Bytes(bytes) => prepared.bind(bytes),
+ };
+ }
+ prepared.execute(pool)
+ },
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<PolledBatch, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.tracking_column();
+ let pk_column = self.primary_key_column();
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ //
+ // An ordering violation is handled one step further. Retrying it would
+ // re-run the same query, re-decode the same rows, and hit the same failure
+ // forever, so the table is latched off instead of being retried each cycle.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ let poll_number = self.polls.fetch_add(1, Ordering::Relaxed) + 1;
+ let poisoned = { self.poisoned_tables.lock().await.clone() };
+ let mut newly_poisoned: Vec<String> = Vec::new();
+
+ for table in &self.config.tables {
+ if poisoned.contains(table) {
+ continue;
+ }
+
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ if !batch.messages.is_empty() {
+ self.cursor_regressions.lock().await.remove(table);
+ }
+ // A full batch means the LIMIT was hit, so more rows are waiting.
+ any_batch_full |= batch.messages.len() as u32 >= batch_size;
+ batches.push(batch);
+ }
+ Err(FetchError::Ordering(reason) | FetchError::MissingTrackingColumn(reason)) => {
+ error!(
+ "Table '{table}' at offset {}: {reason}. Table disabled until restart.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ newly_poisoned.push(table.clone());
+ }
+ Err(FetchError::CursorRegression(reason)) => {
+ // Cleared only by a batch that carried rows, so the run counts polls
+ // since the table last published anything. An idle poll produces as
+ // little as a rejected one does and must not read as recovery.
+ let consecutive = {
+ let mut regressions = self.cursor_regressions.lock().await;
+ let consecutive = regressions.entry(table.clone()).or_insert(0);
+ *consecutive += 1;
+ *consecutive
+ };
+ let offset = last_offset.as_deref().unwrap_or("<start>");
+ if consecutive >= CURSOR_REGRESSION_ERROR_THRESHOLD {
+ error!(
+ "Table '{table}' at offset {offset}: {reason}. Skipping this poll \
+ ({consecutive} consecutive polls)."
+ );
+ } else {
+ warn!("Table '{table}' at offset {offset}: {reason}. Skipping this poll.");
+ }
+ }
+ Err(FetchError::StaleOffsetKind(reason)) => {
+ warn!(
+ "Table '{table}' at offset {}: {reason}. Retrying on the next poll.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::UnusableTrackingValue(reason)) => {
+ error!(
+ "Table '{table}' at offset {}: {reason}. Retrying each poll until the row \
+ is fixed.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::Other(error)) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ if !newly_poisoned.is_empty() {
+ let mut poisoned_tables = self.poisoned_tables.lock().await;
+ poisoned_tables.extend(newly_poisoned);
+ if self
+ .config
+ .tables
+ .iter()
+ .all(|table| poisoned_tables.contains(table))
+ {
+ error!(
+ "All configured tables are disabled because their batches cannot yield a \
+ cursor; {CONNECTOR_NAME} connector with ID: {} is idle until restarted",
+ self.id
+ );
+ }
+ }
+
+ // From the next poll on a disabled table is skipped before anything is logged,
+ // so the one line written when it was disabled can scroll out of the operator's
+ // log window and leave a dark connector looking healthy.
+ if poll_number.is_multiple_of(DISABLED_TABLES_LOG_INTERVAL) {
+ let poisoned_tables = self.poisoned_tables.lock().await;
+ if !poisoned_tables.is_empty() {
+ let mut disabled: Vec<&str> = poisoned_tables.iter().map(String::as_str).collect();
+ disabled.sort_unstable();
+ error!(
+ "{} of {} configured table(s) are still disabled and are not being polled by \
+ {CONNECTOR_NAME} connector with ID: {}, and will stay that way until it is \
+ restarted: {}",
+ disabled.len(),
+ self.config.tables.len(),
+ self.id,
+ disabled.join(", ")
+ );
+ }
+ }
+
+ // Phase 2: stage each table's batch. Nothing touches MySQL or the live state
+ // here: the offsets and the mark/delete are applied in `on_batch_result` once
+ // the runtime has sent the batch and saved this candidate state, and a Nack
+ // throws the candidate away so the same rows are read again.
+ let cleanup_target = self.cleanup_target();
+ let mut candidate_state = self.state.lock().await.clone();
+ let mut operations = Vec::new();
+ let mut messages = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if let Some(target) = &cleanup_target
+ && !batch.messages.is_empty()
+ {
+ if batch.processed_ids.is_empty() {
+ error!(
+ "Table '{}': mark/delete is configured but no primary keys were extracted \
+ from {} row(s), skipping this cycle so rows are not published without \
+ being marked or deleted (check that '{pk_column}' is projected and scalar)",
+ batch.table,
+ batch.messages.len()
+ );
+ continue;
+ }
+
+ match self
+ .cleanup_key_is_unique(pool, &batch.table, &target.key_column)
+ .await
+ {
+ Ok(true) => {}
+ Ok(false) => {
+ error!(
+ "{}. Table disabled until restart.",
+ not_unique_key_message(&batch.table, &target.key_column)
+ );
+ self.poisoned_tables.lock().await.insert(batch.table);
+ continue;
+ }
+ Err(error) => {
+ error!("Table '{}' skipped this cycle: {error}", batch.table);
+ continue;
+ }
+ }
+
+ operations.push(PendingOperation {
+ table: batch.table.clone(),
+ ids: std::mem::take(&mut batch.processed_ids),
+ target: target.clone(),
+ });
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ candidate_state.tracking_offsets.insert(batch.table, offset);
+ }
+ }
+
+ let pending = (total_processed > 0).then(|| {
+ candidate_state.processed_rows += total_processed;
+ candidate_state.pending_operations = operations;
+ PendingBatch {
+ state: candidate_state,
+ acknowledged: false,
+ retiring: false,
+ }
+ });
+
+ self.last_batch_full
+ .store(any_batch_full, Ordering::Relaxed);
+ Ok(PolledBatch { messages, pending })
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, FetchError> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query_kind = self.tracking_kind(table).await;
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.substitute_query_params(custom_query, table, last_offset, batch_size, query_kind)?
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size, query_kind)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ // Settled once for the whole batch rather than per row, since the column list
+ // is a property of the result set.
+ if let Some(first) = rows.first() {
+ ensure_tracking_column_projected(
+ first.columns().iter().map(|column| column.name()),
+ tracking_column,
+ )?;
+ }
+
+ let resume_from = if self.cursor_gates_query() {
+ self.effective_offset(last_offset)
+ } else {
+ None
+ };
+
+ // The result set describes what this query actually returned, so it outranks
+ // whatever `information_schema` said at startup and is the only answer
+ // available for a computed column.
+ let kind = match rows.first().and_then(|row| {
Review Comment:
warning: a late-created `VARCHAR` table with only `0abc` stays unread with `initial_offset = "0"` because empty results leave its type `Unknown`. resolve the type before the filtered query.
##########
core/integration/tests/connectors/fixtures/mysql/source.rs:
##########
@@ -0,0 +1,1774 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::container::{
+ DEFAULT_TEST_STREAM, DEFAULT_TEST_TOPIC, ENV_SOURCE_CONNECTION_STRING, ENV_SOURCE_CUSTOM_QUERY,
+ ENV_SOURCE_DELETE_AFTER_READ, ENV_SOURCE_INCLUDE_METADATA, ENV_SOURCE_INITIAL_OFFSET,
+ ENV_SOURCE_PATH, ENV_SOURCE_PAYLOAD_COLUMN, ENV_SOURCE_PAYLOAD_FORMAT, ENV_SOURCE_PLUGIN_PATH,
+ ENV_SOURCE_POLL_INTERVAL, ENV_SOURCE_PRIMARY_KEY_COLUMN, ENV_SOURCE_PROCESSED_COLUMN,
+ ENV_SOURCE_STREAMS_0_SCHEMA, ENV_SOURCE_STREAMS_0_STREAM, ENV_SOURCE_STREAMS_0_TOPIC,
+ ENV_SOURCE_TABLES, ENV_SOURCE_TRACKING_COLUMN, MySqlContainer, MySqlOps, MySqlSourceOps,
+};
+use async_trait::async_trait;
+use integration::harness::{TestBinaryError, TestFixture};
+use sqlx::{MySql, Pool};
+use std::collections::HashMap;
+
+/// MySQL source fixture for JSON rows with metadata.
+///
+/// Creates a table with typed columns that get serialized as JSON with metadata.
+/// The boolean column is declared `BOOLEAN` (i.e. `tinyint(1)`) so `sqlx-mysql`
+/// reports it as `"BOOLEAN"` and the source emits JSON `true`/`false`, letting the
+/// shared `TestMessage { active: bool }` deserialize unchanged.
+pub struct MySqlSourceJsonFixture {
+ container: MySqlContainer,
+}
+
+impl MySqlOps for MySqlSourceJsonFixture {
+ fn container(&self) -> &MySqlContainer {
+ &self.container
+ }
+}
+
+impl MySqlSourceOps for MySqlSourceJsonFixture {
+ fn table_name(&self) -> &str {
+ Self::TABLE
+ }
+}
+
+impl MySqlSourceJsonFixture {
+ const TABLE: &'static str = "test_messages";
+
+ pub async fn create_table(&self, pool: &Pool<MySql>) {
+ let query = format!(
+ "CREATE TABLE IF NOT EXISTS `{}` (
+ id INT AUTO_INCREMENT PRIMARY KEY,
+ name VARCHAR(255) NOT NULL,
+ `count` INT NOT NULL,
+ amount DOUBLE NOT NULL,
+ active BOOLEAN NOT NULL,
+ `timestamp` BIGINT NOT NULL
+ )",
+ Self::TABLE
+ );
+ sqlx::query(sqlx::AssertSqlSafe(query))
+ .execute(pool)
+ .await
+ .unwrap_or_else(|e| panic!("Failed to create table: {e}"));
+ }
+
+ #[allow(clippy::too_many_arguments)]
+ pub async fn insert_row(
+ &self,
+ pool: &Pool<MySql>,
+ id: i32,
+ name: &str,
+ count: i32,
+ amount: f64,
+ active: bool,
+ timestamp: i64,
+ ) {
+ let query = format!(
+ "INSERT INTO `{}` (id, name, `count`, amount, active, `timestamp`) VALUES (?, ?, ?, ?, ?, ?)",
+ Self::TABLE
+ );
+ sqlx::query(sqlx::AssertSqlSafe(query))
+ .bind(id)
+ .bind(name)
+ .bind(count)
+ .bind(amount)
+ .bind(active)
+ .bind(timestamp)
+ .execute(pool)
+ .await
+ .unwrap_or_else(|e| panic!("Failed to insert row: {e}"));
+ }
+}
+
+#[async_trait]
+impl TestFixture for MySqlSourceJsonFixture {
+ async fn setup() -> Result<Self, TestBinaryError> {
+ let container = MySqlContainer::start().await?;
+ Ok(Self { container })
+ }
+
+ fn connectors_runtime_envs(&self) -> HashMap<String, String> {
Review Comment:
simplification: the fixture methods repeat the same connection and runtime environment fields. move the shared entries into one private helper.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,4071 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source,
+ source::SourceBatchResult, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+/// Upper bound on applying a batch's staged mark/delete after its Ack. The SDK waits
+/// `BATCH_RESULT_TIMEOUT` (30s) for the result, so this has to finish well inside it.
+const ACK_BATCH_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// MySQL rejects a prepared statement with more placeholders than this (error 1390), and
+/// the mark/delete binds one per row of the batch.
+const MAX_CLEANUP_BATCH_SIZE: u32 = 65_535;
+
+/// How many mark/delete statements in a row may fail before the connector stops. The
+/// operations stay in the persisted state and run again on restart.
+const MAX_CONSECUTIVE_CLEANUP_FAILURES: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ pending_batch: Mutex<Option<PendingBatch>>,
+ consecutive_cleanup_failures: AtomicU32,
+ verified_cleanup_keys: Mutex<HashSet<String>>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some("it has too few distinct values to be unique per row");
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some("binary values are compared as base64 text, which does not keep byte order");
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some("MySQL orders these by declaration index, not by label");
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL \
+ or not an ordered scalar (tinyint(1), JSON)"
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "row '{value}' is below the stored offset '{previous}' for tracking_column \
+ '{column}'. Filter the query with `{column} > $offset`"
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. Order the query ascending by '{column}'"
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+ /// Mark/delete work for a batch the runtime has already sent. It is persisted with
+ /// the batch's offsets so a crash between the send and the cleanup reruns it.
+ #[serde(default)]
+ pending_operations: Vec<PendingOperation>,
+}
+
+/// The rows of one table to mark or delete once their batch is acknowledged.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct PendingOperation {
+ table: String,
+ ids: Vec<PkValue>,
+ target: CleanupTarget,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct CleanupTarget {
+ key_column: String,
+ action: CleanupAction,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum CleanupAction {
+ Delete,
+ MarkProcessed { column: String },
+}
+
+/// The candidate state of the batch most recently handed to the runtime. Live state
+/// only takes it over on Ack, so a Nack leaves offsets and rows where they were.
+#[derive(Debug, Clone)]
+struct PendingBatch {
+ state: State,
+ /// The runtime has delivered the batch but its cleanup has not finished yet.
+ acknowledged: bool,
+ /// Cleanup finished and the next poll must persist `state` without the operations,
+ /// so a restart does not run them again.
+ retiring: bool,
+}
+
+struct PolledBatch {
+ messages: Vec<ProducedMessage>,
+ pending: Option<PendingBatch>,
+}
+
+enum CleanupKeyCheck {
+ Unique,
+ NotUnique,
+ TableMissing,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty, list the table even when custom_query hardcodes it"
+ .to_string(),
+ ));
+ }
+
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ if let Some(batch_size) = self.config.batch_size
+ && batch_size > MAX_CLEANUP_BATCH_SIZE
+ && self.cleanup_target().is_some()
+ {
+ return Err(Error::InitError(format!(
+ "batch_size {batch_size} exceeds {MAX_CLEANUP_BATCH_SIZE} with delete_after_read or \
+ processed_column set, MySQL allows at most {MAX_CLEANUP_BATCH_SIZE} placeholders \
+ in the statement that marks or deletes a batch"
+ )));
+ }
+
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "name the column custom_query orders by and filters with $offset"
+ } else {
+ "use a unique, monotonically increasing column"
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set, {guidance}"
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.validate_pending_cleanup()?;
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+ self.validate_cleanup_keys().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ if let Some(state) = self.retiring_state().await {
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: Some(self.serialize_pending_state(&state)?),
+ });
+ }
+
+ // Skip the pacing delay while draining a backlog, but never while a cleanup
+ // retry is pending, or a failing mark/delete would spin without a pause.
+ let cleanup_pending = self.pending_batch.lock().await.is_some();
+ if cleanup_pending || !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ {
+ let pending = self.pending_batch.lock().await;
+ if let Some(pending) = pending.as_ref() {
+ if !pending.acknowledged {
+ error!(
+ "{CONNECTOR_NAME} connector ID: {} was polled while a batch was still in flight",
+ self.id
+ );
+ return Err(Error::InvalidState);
+ }
+ // An empty batch whose Ack retries the cleanup the last one left behind.
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: None,
+ });
+ }
+ }
+
+ let polled = self.poll_tables().await?;
+
+ let processed_rows = match &polled.pending {
+ Some(pending) => pending.state.processed_rows,
+ None => self.state.lock().await.processed_rows,
+ };
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ }
+
+ // Idle cycles stage nothing, so return None to let the runtime skip the fsync.
+ let persisted_state = polled
+ .pending
+ .as_ref()
+ .map(|pending| self.serialize_pending_state(&pending.state))
+ .transpose()?;
+ *self.pending_batch.lock().await = polled.pending;
+
+ Ok(ProducedMessages {
+ schema,
+ messages: polled.messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> {
+ if result == SourceBatchResult::Nack {
+ // Dropping the candidate leaves the live offsets where they were, so the
+ // next poll reads the same rows again. Delivered batches keep their cleanup.
+ let mut pending = self.pending_batch.lock().await;
+ if pending
+ .as_ref()
+ .is_some_and(|pending| !pending.acknowledged && !pending.retiring)
+ {
+ pending.take();
+ }
+ return Ok(());
+ }
+
+ let Some(PendingBatch { mut state, .. }) = self.pending_batch.lock().await.clone() else {
+ return Ok(());
+ };
+
+ let operations = std::mem::take(&mut state.pending_operations);
+ let had_operations = !operations.is_empty();
+ let deadline = tokio::time::Instant::now() + ACK_BATCH_TIMEOUT;
+ let failed_operations = self.apply_pending_operations(operations, deadline).await?;
+
+ if failed_operations.is_empty() {
+ *self.state.lock().await = state.clone();
+ let mut pending = self.pending_batch.lock().await;
+ if had_operations {
+ *pending = Some(PendingBatch {
+ state,
+ acknowledged: false,
+ retiring: true,
+ });
+ } else {
+ pending.take();
+ }
+ } else {
+ state.pending_operations = failed_operations;
+ *self.pending_batch.lock().await = Some(PendingBatch {
+ state,
+ acknowledged: true,
+ retiring: false,
+ });
+ }
+
+ Ok(())
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ let state = restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ pending_operations: Vec::new(),
+ });
+ // The runtime only persists a batch's operations after sending it, so restored
+ // operations belong to a delivered batch and only the cleanup is left to do.
+ let pending_batch = (!state.pending_operations.is_empty()).then(|| PendingBatch {
+ state: state.clone(),
+ acknowledged: true,
+ retiring: false,
+ });
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(state),
+ pending_batch: Mutex::new(pending_batch),
+ consecutive_cleanup_failures: AtomicU32::new(0),
+ verified_cleanup_keys: Mutex::new(HashSet::new()),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ poisoned_tables: Mutex::new(HashSet::new()),
+ tracking_kinds: Mutex::new(HashMap::new()),
+ polls: AtomicU64::new(0),
+ cursor_regressions: Mutex::new(HashMap::new()),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ // `format_offset_value` escapes backslashes, which under NO_BACKSLASH_ESCAPES
+ // would reach MySQL as literal characters and filter on the wrong cursor.
+ .after_connect(|connection, _| {
+ Box::pin(async move {
+ sqlx::query(
+ "SET SESSION sql_mode = REPLACE(@@SESSION.sql_mode, 'NO_BACKSLASH_ESCAPES', '')",
+ )
+ .execute(&mut *connection)
+ .await?;
+ Ok(())
+ })
+ })
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn tracking_column(&self) -> &str {
+ self.config.tracking_column.as_deref().unwrap_or("id")
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn resolve_tracking_columns(&self) -> Result<HashMap<String, OffsetKind>, Error> {
+ // Whether the connector owns the query, and can therefore hold the schema to
+ // what that query will do with it.
+ let strict = self.config.custom_query.is_none();
+ let pool = self.get_pool()?;
+ let tracking_column = self.tracking_column();
+ let mut kinds = HashMap::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ }
+
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ }
+
+ warn!(
+ "tracking_column '{tracking_column}' is not a column of table '{table}'; \
+ assuming custom_query projects it. Its comparison order is resolved from \
+ the first result set instead."
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ // The column's type survives a plain projection, so it is worth rejecting
+ // whatever the query looks like. A custom_query aliasing an expression onto
+ // a name the table also uses is the one false positive, hence the hint.
+ if let Some(reason) = unusable_tracking_type(&data_type) {
+ let aliasing_hint = if self.config.custom_query.is_some() {
+ " If custom_query aliases a different expression to this name, rename the \
+ alias so it does not collide with the column."
+ } else {
+ ""
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': {reason}. Use an integer, timestamp, or string \
+ column.{aliasing_hint}"
+ )));
+ }
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; a row \
+ whose value is NULL cannot produce a cursor and will fail that table's batch. \
+ Have custom_query exclude those rows, or declare the column NOT NULL."
+ );
+ }
+
+ if is_collation_ordered(&data_type) {
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': values must be unique and increase in its collation order \
+ (zero-pad numbers, prefer a _bin collation)"
+ );
+ }
+
+ kinds.insert(table.clone(), offset_kind_for_data_type(&data_type));
+ }
+ Ok(kinds)
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
Review Comment:
simplification: `serialize_state()` only forwards one caller to `ConnectorState::serialize()`. call `ConnectorState::serialize()` directly from `serialize_pending_state()` and remove the wrapper.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,4071 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source,
+ source::SourceBatchResult, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+/// Upper bound on applying a batch's staged mark/delete after its Ack. The SDK waits
+/// `BATCH_RESULT_TIMEOUT` (30s) for the result, so this has to finish well inside it.
+const ACK_BATCH_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// MySQL rejects a prepared statement with more placeholders than this (error 1390), and
+/// the mark/delete binds one per row of the batch.
+const MAX_CLEANUP_BATCH_SIZE: u32 = 65_535;
+
+/// How many mark/delete statements in a row may fail before the connector stops. The
+/// operations stay in the persisted state and run again on restart.
+const MAX_CONSECUTIVE_CLEANUP_FAILURES: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ pending_batch: Mutex<Option<PendingBatch>>,
+ consecutive_cleanup_failures: AtomicU32,
+ verified_cleanup_keys: Mutex<HashSet<String>>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some("it has too few distinct values to be unique per row");
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some("binary values are compared as base64 text, which does not keep byte order");
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some("MySQL orders these by declaration index, not by label");
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL \
+ or not an ordered scalar (tinyint(1), JSON)"
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "row '{value}' is below the stored offset '{previous}' for tracking_column \
+ '{column}'. Filter the query with `{column} > $offset`"
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. Order the query ascending by '{column}'"
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+ /// Mark/delete work for a batch the runtime has already sent. It is persisted with
+ /// the batch's offsets so a crash between the send and the cleanup reruns it.
+ #[serde(default)]
+ pending_operations: Vec<PendingOperation>,
+}
+
+/// The rows of one table to mark or delete once their batch is acknowledged.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct PendingOperation {
+ table: String,
+ ids: Vec<PkValue>,
+ target: CleanupTarget,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct CleanupTarget {
+ key_column: String,
+ action: CleanupAction,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum CleanupAction {
+ Delete,
+ MarkProcessed { column: String },
+}
+
+/// The candidate state of the batch most recently handed to the runtime. Live state
+/// only takes it over on Ack, so a Nack leaves offsets and rows where they were.
+#[derive(Debug, Clone)]
+struct PendingBatch {
+ state: State,
+ /// The runtime has delivered the batch but its cleanup has not finished yet.
+ acknowledged: bool,
+ /// Cleanup finished and the next poll must persist `state` without the operations,
+ /// so a restart does not run them again.
+ retiring: bool,
+}
+
+struct PolledBatch {
+ messages: Vec<ProducedMessage>,
+ pending: Option<PendingBatch>,
+}
+
+enum CleanupKeyCheck {
+ Unique,
+ NotUnique,
+ TableMissing,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty, list the table even when custom_query hardcodes it"
+ .to_string(),
+ ));
+ }
+
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ if let Some(batch_size) = self.config.batch_size
+ && batch_size > MAX_CLEANUP_BATCH_SIZE
+ && self.cleanup_target().is_some()
+ {
+ return Err(Error::InitError(format!(
+ "batch_size {batch_size} exceeds {MAX_CLEANUP_BATCH_SIZE} with delete_after_read or \
+ processed_column set, MySQL allows at most {MAX_CLEANUP_BATCH_SIZE} placeholders \
+ in the statement that marks or deletes a batch"
+ )));
+ }
+
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "name the column custom_query orders by and filters with $offset"
+ } else {
+ "use a unique, monotonically increasing column"
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set, {guidance}"
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.validate_pending_cleanup()?;
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+ self.validate_cleanup_keys().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ if let Some(state) = self.retiring_state().await {
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: Some(self.serialize_pending_state(&state)?),
+ });
+ }
+
+ // Skip the pacing delay while draining a backlog, but never while a cleanup
+ // retry is pending, or a failing mark/delete would spin without a pause.
+ let cleanup_pending = self.pending_batch.lock().await.is_some();
+ if cleanup_pending || !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ {
+ let pending = self.pending_batch.lock().await;
+ if let Some(pending) = pending.as_ref() {
+ if !pending.acknowledged {
+ error!(
+ "{CONNECTOR_NAME} connector ID: {} was polled while a batch was still in flight",
+ self.id
+ );
+ return Err(Error::InvalidState);
+ }
+ // An empty batch whose Ack retries the cleanup the last one left behind.
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: None,
+ });
+ }
+ }
+
+ let polled = self.poll_tables().await?;
+
+ let processed_rows = match &polled.pending {
+ Some(pending) => pending.state.processed_rows,
+ None => self.state.lock().await.processed_rows,
+ };
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ }
+
+ // Idle cycles stage nothing, so return None to let the runtime skip the fsync.
+ let persisted_state = polled
+ .pending
+ .as_ref()
+ .map(|pending| self.serialize_pending_state(&pending.state))
+ .transpose()?;
+ *self.pending_batch.lock().await = polled.pending;
+
+ Ok(ProducedMessages {
+ schema,
+ messages: polled.messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> {
+ if result == SourceBatchResult::Nack {
+ // Dropping the candidate leaves the live offsets where they were, so the
+ // next poll reads the same rows again. Delivered batches keep their cleanup.
+ let mut pending = self.pending_batch.lock().await;
+ if pending
+ .as_ref()
+ .is_some_and(|pending| !pending.acknowledged && !pending.retiring)
+ {
+ pending.take();
+ }
+ return Ok(());
+ }
+
+ let Some(PendingBatch { mut state, .. }) = self.pending_batch.lock().await.clone() else {
+ return Ok(());
+ };
+
+ let operations = std::mem::take(&mut state.pending_operations);
+ let had_operations = !operations.is_empty();
+ let deadline = tokio::time::Instant::now() + ACK_BATCH_TIMEOUT;
+ let failed_operations = self.apply_pending_operations(operations, deadline).await?;
+
+ if failed_operations.is_empty() {
+ *self.state.lock().await = state.clone();
+ let mut pending = self.pending_batch.lock().await;
+ if had_operations {
+ *pending = Some(PendingBatch {
+ state,
+ acknowledged: false,
+ retiring: true,
+ });
+ } else {
+ pending.take();
+ }
+ } else {
+ state.pending_operations = failed_operations;
+ *self.pending_batch.lock().await = Some(PendingBatch {
+ state,
+ acknowledged: true,
+ retiring: false,
+ });
+ }
+
+ Ok(())
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ let state = restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ pending_operations: Vec::new(),
+ });
+ // The runtime only persists a batch's operations after sending it, so restored
+ // operations belong to a delivered batch and only the cleanup is left to do.
+ let pending_batch = (!state.pending_operations.is_empty()).then(|| PendingBatch {
+ state: state.clone(),
+ acknowledged: true,
+ retiring: false,
+ });
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(state),
+ pending_batch: Mutex::new(pending_batch),
+ consecutive_cleanup_failures: AtomicU32::new(0),
+ verified_cleanup_keys: Mutex::new(HashSet::new()),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ poisoned_tables: Mutex::new(HashSet::new()),
+ tracking_kinds: Mutex::new(HashMap::new()),
+ polls: AtomicU64::new(0),
+ cursor_regressions: Mutex::new(HashMap::new()),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ // `format_offset_value` escapes backslashes, which under NO_BACKSLASH_ESCAPES
+ // would reach MySQL as literal characters and filter on the wrong cursor.
+ .after_connect(|connection, _| {
+ Box::pin(async move {
+ sqlx::query(
+ "SET SESSION sql_mode = REPLACE(@@SESSION.sql_mode, 'NO_BACKSLASH_ESCAPES', '')",
+ )
+ .execute(&mut *connection)
+ .await?;
+ Ok(())
+ })
+ })
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn tracking_column(&self) -> &str {
+ self.config.tracking_column.as_deref().unwrap_or("id")
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn resolve_tracking_columns(&self) -> Result<HashMap<String, OffsetKind>, Error> {
+ // Whether the connector owns the query, and can therefore hold the schema to
+ // what that query will do with it.
+ let strict = self.config.custom_query.is_none();
+ let pool = self.get_pool()?;
+ let tracking_column = self.tracking_column();
+ let mut kinds = HashMap::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ }
+
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ }
+
+ warn!(
+ "tracking_column '{tracking_column}' is not a column of table '{table}'; \
+ assuming custom_query projects it. Its comparison order is resolved from \
+ the first result set instead."
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ // The column's type survives a plain projection, so it is worth rejecting
+ // whatever the query looks like. A custom_query aliasing an expression onto
+ // a name the table also uses is the one false positive, hence the hint.
+ if let Some(reason) = unusable_tracking_type(&data_type) {
+ let aliasing_hint = if self.config.custom_query.is_some() {
+ " If custom_query aliases a different expression to this name, rename the \
+ alias so it does not collide with the column."
+ } else {
+ ""
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': {reason}. Use an integer, timestamp, or string \
+ column.{aliasing_hint}"
+ )));
+ }
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; a row \
+ whose value is NULL cannot produce a cursor and will fail that table's batch. \
+ Have custom_query exclude those rows, or declare the column NOT NULL."
+ );
+ }
+
+ if is_collation_ordered(&data_type) {
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': values must be unique and increase in its collation order \
+ (zero-pad numbers, prefer a _bin collation)"
+ );
+ }
+
+ kinds.insert(table.clone(), offset_kind_for_data_type(&data_type));
+ }
+ Ok(kinds)
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn serialize_pending_state(&self, state: &State) -> Result<ConnectorState, Error> {
+ self.serialize_state(state).ok_or_else(|| {
+ Error::Serialization(format!("failed to serialize {CONNECTOR_NAME} state"))
+ })
+ }
+
+ async fn retiring_state(&self) -> Option<State> {
+ self.pending_batch
+ .lock()
+ .await
+ .as_ref()
+ .filter(|pending| pending.retiring)
+ .map(|pending| pending.state.clone())
+ }
+
+ fn primary_key_column(&self) -> &str {
+ self.config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(self.tracking_column())
+ }
+
+ fn cleanup_target(&self) -> Option<CleanupTarget> {
+ let action = if self.config.delete_after_read.unwrap_or(false) {
+ CleanupAction::Delete
+ } else if let Some(column) = &self.config.processed_column {
+ CleanupAction::MarkProcessed {
+ column: column.clone(),
+ }
+ } else {
+ return None;
+ };
+ Some(CleanupTarget {
+ key_column: self.primary_key_column().to_string(),
+ action,
+ })
+ }
+
+ /// Restored operations were staged under the config of the run that wrote them.
+ /// Running them under a different key or action would touch rows nobody read.
+ fn validate_pending_cleanup(&mut self) -> Result<(), Error> {
+ let configured = self.cleanup_target();
+ for operation in &self.state.get_mut().pending_operations {
+ if configured.as_ref() != Some(&operation.target) {
+ return Err(Error::InitError(format!(
+ "the cleanup configuration changed while cleanup of {} delivered row(s) in \
+ table '{}' is pending; restore primary_key_column '{}' and action {:?}",
+ operation.ids.len(),
+ operation.table,
+ operation.target.key_column,
+ operation.target.action
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ /// `WHERE key IN (...)` on a key that repeats marks or deletes every row sharing a
+ /// value, including ones that were never read. Tables that don't exist yet are
+ /// checked by the first poll that stages cleanup for them.
+ async fn validate_cleanup_keys(&mut self) -> Result<(), Error> {
+ let Some(target) = self.cleanup_target() else {
+ return Ok(());
+ };
+ let pool = self.get_pool()?;
+ let mut verified = HashSet::with_capacity(self.config.tables.len());
+ for table in &self.config.tables {
+ let check = cleanup_key_check(pool, table, &target.key_column)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to check primary_key_column '{}' on table '{table}': {e}",
+ target.key_column
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ verified.insert(table.clone());
+ }
+ CleanupKeyCheck::NotUnique => {
+ return Err(Error::InitError(not_unique_key_message(
+ table,
+ &target.key_column,
+ )));
+ }
+ CleanupKeyCheck::TableMissing => {}
+ }
+ }
+ *self.verified_cleanup_keys.get_mut() = verified;
+ Ok(())
+ }
+
+ async fn cleanup_key_is_unique(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ key_column: &str,
+ ) -> Result<bool, Error> {
+ if self.verified_cleanup_keys.lock().await.contains(table) {
+ return Ok(true);
+ }
+ let check = cleanup_key_check(pool, table, key_column)
+ .await
+ .map_err(|e| {
+ Error::Connection(format!(
+ "failed to check primary_key_column '{key_column}' on table '{table}': {e}"
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ self.verified_cleanup_keys
+ .lock()
+ .await
+ .insert(table.to_string());
+ Ok(true)
+ }
+ CleanupKeyCheck::NotUnique => Ok(false),
+ CleanupKeyCheck::TableMissing => Err(Error::Connection(format!(
+ "table '{table}' disappeared before its primary_key_column could be checked"
+ ))),
+ }
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ /// Best current knowledge of how a table's tracking values compare.
+ async fn tracking_kind(&self, table: &str) -> OffsetKind {
+ self.tracking_kinds
+ .lock()
+ .await
+ .get(table)
+ .copied()
+ .unwrap_or(OffsetKind::Unknown)
+ }
+
+ /// Whether the stored cursor is what limits the rows a poll can see. The built
+ /// polling query always filters on it; a `custom_query` does so only if it
+ /// interpolates `$offset`.
+ fn cursor_gates_query(&self) -> bool {
+ match &self.config.custom_query {
+ Some(query) => query.contains("$offset"),
+ None => true,
+ }
+ }
+
+ /// The offset a poll actually resumes from: the stored cursor, or the configured
+ /// starting point before anything has been stored.
+ fn effective_offset(&self, last_offset: &Option<String>) -> Option<String> {
+ last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let offset_value = self.effective_offset(last_offset).unwrap_or_default();
+ let offset = format_offset_value(&offset_value, kind);
+ let quoted_table = quote_qualified_identifier(table)?;
+ let limit = batch_size.to_string();
+ let now = Utc::now();
+ let now_unix = now.timestamp().to_string();
+ let now = now.to_rfc3339();
+ let replacements = [
+ ("$table", quoted_table.as_str()),
+ ("$offset", offset.as_str()),
+ ("$limit", limit.as_str()),
+ ("$now_unix", now_unix.as_str()),
+ ("$now", now.as_str()),
+ ];
+
+ Ok(substitute_query_tokens(query, &replacements))
+ }
+
+ /// The query is substituted as text and re-executed once per configured table on
+ /// every poll for the life of the process, so a shape that cannot terminate does
+ /// not fail, it repeats. What can be judged from the text alone is judged here,
+ /// at startup, rather than left to the first poll.
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let tracking_column = self.tracking_column();
+
+ match leading_statement_token(query) {
+ None => {
+ return Err(Error::InitError(
+ "custom_query has no statement: it is empty, or nothing but comments"
+ .to_string(),
+ ));
+ }
+ Some(token) if !READ_STATEMENT_STARTS.contains(&token.as_str()) => {
+ return Err(Error::InitError(format!(
+ "custom_query must be a read starting with SELECT, WITH or TABLE, found \
+ '{token}'"
+ )));
+ }
+ Some(_) => {}
+ }
+
+ // Collapsed so `ORDER\nBY` and `ORDER BY` read the same as `ORDER BY`; the
+ // check hard-fails, so a formatting difference must not decide it.
+ let normalized = query
+ .split_whitespace()
+ .collect::<Vec<_>>()
+ .join(" ")
+ .to_uppercase();
+
+ if query.contains("$offset") {
+ if !normalized.contains("ORDER BY") {
+ return Err(Error::InitError(format!(
+ "custom_query filters on $offset but has no ORDER BY, add \
+ `ORDER BY {tracking_column}`"
+ )));
+ }
+ } else if !self.config.delete_after_read.unwrap_or(false)
+ && self.config.processed_column.is_none()
+ {
+ return Err(Error::InitError(format!(
+ "custom_query must filter with $offset (`WHERE {tracking_column} > $offset`), or \
+ set delete_after_read or processed_column"
+ )));
+ }
+
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "custom_query uses $table but no tables are configured".to_string(),
+ ));
+ }
+ // Without $table the same SQL is executed once per table, so identical rows
+ // would be emitted repeatedly, each carrying a different table's metadata,
+ // message ID, and tracking offset. Table identity is not recoverable from a
+ // static query, so reject the combination instead of guessing one.
+ if !query.contains("$table") && self.config.tables.len() > 1 {
+ return Err(Error::InitError(format!(
+ "custom_query has no $table placeholder but {} tables are configured ({}), use \
+ $table in the query or configure a single table",
+ self.config.tables.len(),
+ self.config.tables.join(", ")
+ )));
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = self.effective_offset(last_offset) {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(&offset, kind)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn apply_pending_operations(
+ &self,
+ operations: Vec<PendingOperation>,
+ deadline: tokio::time::Instant,
+ ) -> Result<Vec<PendingOperation>, Error> {
+ let mut failed_operations = Vec::new();
+ let mut operations = operations.into_iter();
+ while let Some(operation) = operations.next() {
+ if tokio::time::Instant::now() >= deadline {
+ failed_operations.push(operation);
+ failed_operations.extend(operations);
+ break;
+ }
+ let result = tokio::time::timeout_at(deadline, async {
+ let pool = self.get_pool()?;
+ self.mark_or_delete_processed_rows(pool, &operation).await
+ })
+ .await;
+ match result {
+ Ok(Ok(())) => {
+ self.consecutive_cleanup_failures
+ .store(0, Ordering::Relaxed);
+ }
+ Ok(Err(error)) => {
+ self.record_cleanup_failure(&operation, &error)?;
+ failed_operations.push(operation);
+ }
+ Err(_) => {
+ self.record_cleanup_failure(
+ &operation,
+ "cleanup exceeded the 10s budget for one Ack",
+ )?;
+ failed_operations.push(operation);
+ }
+ }
+ }
+ Ok(failed_operations)
+ }
+
+ fn record_cleanup_failure(
+ &self,
+ operation: &PendingOperation,
+ error: impl std::fmt::Display,
+ ) -> Result<(), Error> {
+ let consecutive_failures = self
+ .consecutive_cleanup_failures
+ .fetch_add(1, Ordering::Relaxed)
+ .saturating_add(1);
+ error!(
+ "Failed to mark or delete {} delivered row(s) in table '{}' for {CONNECTOR_NAME} \
+ connector ID: {}, retrying on the next poll ({consecutive_failures} consecutive \
+ failures): {error}",
+ operation.ids.len(),
+ operation.table,
+ self.id
+ );
+ if consecutive_failures >= MAX_CONSECUTIVE_CLEANUP_FAILURES {
+ return Err(Error::Connection(format!(
+ "stopping {CONNECTOR_NAME} connector ID {} after {consecutive_failures} \
+ consecutive mark/delete failures; the cleanup stays in the saved state and \
+ runs again on restart",
+ self.id
+ )));
+ }
+ Ok(())
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ operation: &PendingOperation,
+ ) -> Result<(), Error> {
+ let PendingOperation { table, ids, target } = operation;
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(&target.key_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = match &target.action {
+ CleanupAction::Delete => {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
+ }
+ CleanupAction::MarkProcessed { column } => {
+ let quoted_processed = quote_identifier(column)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({placeholders})"
+ )
+ }
+ };
+
+ // The query and its binds are rebuilt on every attempt because with_retry
+ // calls the closure once per try and each `execute` consumes the query.
+ with_retry(
+ || {
+ let mut prepared = sqlx::query(sqlx::AssertSqlSafe(query.as_str()));
+ for id in ids {
+ prepared = match id {
+ PkValue::Text(text) => prepared.bind(text),
+ PkValue::Bytes(bytes) => prepared.bind(bytes),
+ };
+ }
+ prepared.execute(pool)
+ },
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<PolledBatch, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.tracking_column();
+ let pk_column = self.primary_key_column();
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ //
+ // An ordering violation is handled one step further. Retrying it would
+ // re-run the same query, re-decode the same rows, and hit the same failure
+ // forever, so the table is latched off instead of being retried each cycle.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ let poll_number = self.polls.fetch_add(1, Ordering::Relaxed) + 1;
+ let poisoned = { self.poisoned_tables.lock().await.clone() };
+ let mut newly_poisoned: Vec<String> = Vec::new();
+
+ for table in &self.config.tables {
+ if poisoned.contains(table) {
+ continue;
+ }
+
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ if !batch.messages.is_empty() {
+ self.cursor_regressions.lock().await.remove(table);
+ }
+ // A full batch means the LIMIT was hit, so more rows are waiting.
+ any_batch_full |= batch.messages.len() as u32 >= batch_size;
+ batches.push(batch);
+ }
+ Err(FetchError::Ordering(reason) | FetchError::MissingTrackingColumn(reason)) => {
+ error!(
+ "Table '{table}' at offset {}: {reason}. Table disabled until restart.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ newly_poisoned.push(table.clone());
+ }
+ Err(FetchError::CursorRegression(reason)) => {
+ // Cleared only by a batch that carried rows, so the run counts polls
+ // since the table last published anything. An idle poll produces as
+ // little as a rejected one does and must not read as recovery.
+ let consecutive = {
+ let mut regressions = self.cursor_regressions.lock().await;
+ let consecutive = regressions.entry(table.clone()).or_insert(0);
+ *consecutive += 1;
+ *consecutive
+ };
+ let offset = last_offset.as_deref().unwrap_or("<start>");
+ if consecutive >= CURSOR_REGRESSION_ERROR_THRESHOLD {
+ error!(
+ "Table '{table}' at offset {offset}: {reason}. Skipping this poll \
+ ({consecutive} consecutive polls)."
+ );
+ } else {
+ warn!("Table '{table}' at offset {offset}: {reason}. Skipping this poll.");
+ }
+ }
+ Err(FetchError::StaleOffsetKind(reason)) => {
+ warn!(
+ "Table '{table}' at offset {}: {reason}. Retrying on the next poll.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::UnusableTrackingValue(reason)) => {
+ error!(
+ "Table '{table}' at offset {}: {reason}. Retrying each poll until the row \
+ is fixed.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::Other(error)) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ if !newly_poisoned.is_empty() {
+ let mut poisoned_tables = self.poisoned_tables.lock().await;
+ poisoned_tables.extend(newly_poisoned);
+ if self
+ .config
+ .tables
+ .iter()
+ .all(|table| poisoned_tables.contains(table))
+ {
+ error!(
+ "All configured tables are disabled because their batches cannot yield a \
+ cursor; {CONNECTOR_NAME} connector with ID: {} is idle until restarted",
+ self.id
+ );
+ }
+ }
+
+ // From the next poll on a disabled table is skipped before anything is logged,
+ // so the one line written when it was disabled can scroll out of the operator's
+ // log window and leave a dark connector looking healthy.
+ if poll_number.is_multiple_of(DISABLED_TABLES_LOG_INTERVAL) {
+ let poisoned_tables = self.poisoned_tables.lock().await;
+ if !poisoned_tables.is_empty() {
+ let mut disabled: Vec<&str> = poisoned_tables.iter().map(String::as_str).collect();
+ disabled.sort_unstable();
+ error!(
+ "{} of {} configured table(s) are still disabled and are not being polled by \
+ {CONNECTOR_NAME} connector with ID: {}, and will stay that way until it is \
+ restarted: {}",
+ disabled.len(),
+ self.config.tables.len(),
+ self.id,
+ disabled.join(", ")
+ );
+ }
+ }
+
+ // Phase 2: stage each table's batch. Nothing touches MySQL or the live state
+ // here: the offsets and the mark/delete are applied in `on_batch_result` once
+ // the runtime has sent the batch and saved this candidate state, and a Nack
+ // throws the candidate away so the same rows are read again.
+ let cleanup_target = self.cleanup_target();
+ let mut candidate_state = self.state.lock().await.clone();
+ let mut operations = Vec::new();
+ let mut messages = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if let Some(target) = &cleanup_target
+ && !batch.messages.is_empty()
+ {
+ if batch.processed_ids.is_empty() {
+ error!(
+ "Table '{}': mark/delete is configured but no primary keys were extracted \
+ from {} row(s), skipping this cycle so rows are not published without \
+ being marked or deleted (check that '{pk_column}' is projected and scalar)",
+ batch.table,
+ batch.messages.len()
+ );
+ continue;
+ }
+
+ match self
+ .cleanup_key_is_unique(pool, &batch.table, &target.key_column)
+ .await
+ {
+ Ok(true) => {}
+ Ok(false) => {
+ error!(
+ "{}. Table disabled until restart.",
+ not_unique_key_message(&batch.table, &target.key_column)
+ );
+ self.poisoned_tables.lock().await.insert(batch.table);
+ continue;
+ }
+ Err(error) => {
+ error!("Table '{}' skipped this cycle: {error}", batch.table);
+ continue;
+ }
+ }
+
+ operations.push(PendingOperation {
+ table: batch.table.clone(),
+ ids: std::mem::take(&mut batch.processed_ids),
+ target: target.clone(),
+ });
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ candidate_state.tracking_offsets.insert(batch.table, offset);
+ }
+ }
+
+ let pending = (total_processed > 0).then(|| {
+ candidate_state.processed_rows += total_processed;
+ candidate_state.pending_operations = operations;
+ PendingBatch {
+ state: candidate_state,
+ acknowledged: false,
+ retiring: false,
+ }
+ });
+
+ self.last_batch_full
+ .store(any_batch_full, Ordering::Relaxed);
+ Ok(PolledBatch { messages, pending })
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, FetchError> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query_kind = self.tracking_kind(table).await;
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.substitute_query_params(custom_query, table, last_offset, batch_size, query_kind)?
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size, query_kind)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
Review Comment:
warning: literal cursor values create a different cached statement on each advance, so later polls cannot reuse it. disable persistence or bind cursor values in a reusable statement.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,4071 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source,
+ source::SourceBatchResult, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+/// Upper bound on applying a batch's staged mark/delete after its Ack. The SDK waits
+/// `BATCH_RESULT_TIMEOUT` (30s) for the result, so this has to finish well inside it.
+const ACK_BATCH_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// MySQL rejects a prepared statement with more placeholders than this (error 1390), and
+/// the mark/delete binds one per row of the batch.
+const MAX_CLEANUP_BATCH_SIZE: u32 = 65_535;
+
+/// How many mark/delete statements in a row may fail before the connector stops. The
+/// operations stay in the persisted state and run again on restart.
+const MAX_CONSECUTIVE_CLEANUP_FAILURES: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ pending_batch: Mutex<Option<PendingBatch>>,
+ consecutive_cleanup_failures: AtomicU32,
+ verified_cleanup_keys: Mutex<HashSet<String>>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some("it has too few distinct values to be unique per row");
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some("binary values are compared as base64 text, which does not keep byte order");
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some("MySQL orders these by declaration index, not by label");
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL \
+ or not an ordered scalar (tinyint(1), JSON)"
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "row '{value}' is below the stored offset '{previous}' for tracking_column \
+ '{column}'. Filter the query with `{column} > $offset`"
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. Order the query ascending by '{column}'"
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+ /// Mark/delete work for a batch the runtime has already sent. It is persisted with
+ /// the batch's offsets so a crash between the send and the cleanup reruns it.
+ #[serde(default)]
+ pending_operations: Vec<PendingOperation>,
+}
+
+/// The rows of one table to mark or delete once their batch is acknowledged.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct PendingOperation {
+ table: String,
+ ids: Vec<PkValue>,
+ target: CleanupTarget,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct CleanupTarget {
+ key_column: String,
+ action: CleanupAction,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum CleanupAction {
+ Delete,
+ MarkProcessed { column: String },
+}
+
+/// The candidate state of the batch most recently handed to the runtime. Live state
+/// only takes it over on Ack, so a Nack leaves offsets and rows where they were.
+#[derive(Debug, Clone)]
+struct PendingBatch {
+ state: State,
+ /// The runtime has delivered the batch but its cleanup has not finished yet.
+ acknowledged: bool,
+ /// Cleanup finished and the next poll must persist `state` without the operations,
+ /// so a restart does not run them again.
+ retiring: bool,
+}
+
+struct PolledBatch {
+ messages: Vec<ProducedMessage>,
+ pending: Option<PendingBatch>,
+}
+
+enum CleanupKeyCheck {
+ Unique,
+ NotUnique,
+ TableMissing,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty, list the table even when custom_query hardcodes it"
+ .to_string(),
+ ));
+ }
+
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ if let Some(batch_size) = self.config.batch_size
+ && batch_size > MAX_CLEANUP_BATCH_SIZE
+ && self.cleanup_target().is_some()
+ {
+ return Err(Error::InitError(format!(
+ "batch_size {batch_size} exceeds {MAX_CLEANUP_BATCH_SIZE} with delete_after_read or \
+ processed_column set, MySQL allows at most {MAX_CLEANUP_BATCH_SIZE} placeholders \
+ in the statement that marks or deletes a batch"
+ )));
+ }
+
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "name the column custom_query orders by and filters with $offset"
+ } else {
+ "use a unique, monotonically increasing column"
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set, {guidance}"
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.validate_pending_cleanup()?;
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+ self.validate_cleanup_keys().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ if let Some(state) = self.retiring_state().await {
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: Some(self.serialize_pending_state(&state)?),
+ });
+ }
+
+ // Skip the pacing delay while draining a backlog, but never while a cleanup
+ // retry is pending, or a failing mark/delete would spin without a pause.
+ let cleanup_pending = self.pending_batch.lock().await.is_some();
+ if cleanup_pending || !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ {
+ let pending = self.pending_batch.lock().await;
+ if let Some(pending) = pending.as_ref() {
+ if !pending.acknowledged {
+ error!(
+ "{CONNECTOR_NAME} connector ID: {} was polled while a batch was still in flight",
+ self.id
+ );
+ return Err(Error::InvalidState);
+ }
+ // An empty batch whose Ack retries the cleanup the last one left behind.
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: None,
+ });
+ }
+ }
+
+ let polled = self.poll_tables().await?;
+
+ let processed_rows = match &polled.pending {
+ Some(pending) => pending.state.processed_rows,
+ None => self.state.lock().await.processed_rows,
+ };
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ }
+
+ // Idle cycles stage nothing, so return None to let the runtime skip the fsync.
+ let persisted_state = polled
+ .pending
+ .as_ref()
+ .map(|pending| self.serialize_pending_state(&pending.state))
+ .transpose()?;
+ *self.pending_batch.lock().await = polled.pending;
+
+ Ok(ProducedMessages {
+ schema,
+ messages: polled.messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> {
+ if result == SourceBatchResult::Nack {
+ // Dropping the candidate leaves the live offsets where they were, so the
+ // next poll reads the same rows again. Delivered batches keep their cleanup.
+ let mut pending = self.pending_batch.lock().await;
+ if pending
+ .as_ref()
+ .is_some_and(|pending| !pending.acknowledged && !pending.retiring)
+ {
+ pending.take();
+ }
+ return Ok(());
+ }
+
+ let Some(PendingBatch { mut state, .. }) = self.pending_batch.lock().await.clone() else {
+ return Ok(());
+ };
+
+ let operations = std::mem::take(&mut state.pending_operations);
+ let had_operations = !operations.is_empty();
+ let deadline = tokio::time::Instant::now() + ACK_BATCH_TIMEOUT;
+ let failed_operations = self.apply_pending_operations(operations, deadline).await?;
+
+ if failed_operations.is_empty() {
+ *self.state.lock().await = state.clone();
+ let mut pending = self.pending_batch.lock().await;
+ if had_operations {
+ *pending = Some(PendingBatch {
+ state,
+ acknowledged: false,
+ retiring: true,
+ });
+ } else {
+ pending.take();
+ }
+ } else {
+ state.pending_operations = failed_operations;
+ *self.pending_batch.lock().await = Some(PendingBatch {
+ state,
+ acknowledged: true,
+ retiring: false,
+ });
+ }
+
+ Ok(())
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ let state = restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ pending_operations: Vec::new(),
+ });
+ // The runtime only persists a batch's operations after sending it, so restored
+ // operations belong to a delivered batch and only the cleanup is left to do.
+ let pending_batch = (!state.pending_operations.is_empty()).then(|| PendingBatch {
+ state: state.clone(),
+ acknowledged: true,
+ retiring: false,
+ });
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(state),
+ pending_batch: Mutex::new(pending_batch),
+ consecutive_cleanup_failures: AtomicU32::new(0),
+ verified_cleanup_keys: Mutex::new(HashSet::new()),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ poisoned_tables: Mutex::new(HashSet::new()),
+ tracking_kinds: Mutex::new(HashMap::new()),
+ polls: AtomicU64::new(0),
+ cursor_regressions: Mutex::new(HashMap::new()),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ // `format_offset_value` escapes backslashes, which under NO_BACKSLASH_ESCAPES
+ // would reach MySQL as literal characters and filter on the wrong cursor.
+ .after_connect(|connection, _| {
+ Box::pin(async move {
+ sqlx::query(
+ "SET SESSION sql_mode = REPLACE(@@SESSION.sql_mode, 'NO_BACKSLASH_ESCAPES', '')",
+ )
+ .execute(&mut *connection)
+ .await?;
+ Ok(())
+ })
+ })
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn tracking_column(&self) -> &str {
+ self.config.tracking_column.as_deref().unwrap_or("id")
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn resolve_tracking_columns(&self) -> Result<HashMap<String, OffsetKind>, Error> {
+ // Whether the connector owns the query, and can therefore hold the schema to
+ // what that query will do with it.
+ let strict = self.config.custom_query.is_none();
+ let pool = self.get_pool()?;
+ let tracking_column = self.tracking_column();
+ let mut kinds = HashMap::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ }
+
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ }
+
+ warn!(
+ "tracking_column '{tracking_column}' is not a column of table '{table}'; \
+ assuming custom_query projects it. Its comparison order is resolved from \
+ the first result set instead."
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ // The column's type survives a plain projection, so it is worth rejecting
+ // whatever the query looks like. A custom_query aliasing an expression onto
+ // a name the table also uses is the one false positive, hence the hint.
+ if let Some(reason) = unusable_tracking_type(&data_type) {
+ let aliasing_hint = if self.config.custom_query.is_some() {
+ " If custom_query aliases a different expression to this name, rename the \
+ alias so it does not collide with the column."
+ } else {
+ ""
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': {reason}. Use an integer, timestamp, or string \
+ column.{aliasing_hint}"
+ )));
+ }
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; a row \
+ whose value is NULL cannot produce a cursor and will fail that table's batch. \
+ Have custom_query exclude those rows, or declare the column NOT NULL."
+ );
+ }
+
+ if is_collation_ordered(&data_type) {
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': values must be unique and increase in its collation order \
+ (zero-pad numbers, prefer a _bin collation)"
+ );
+ }
+
+ kinds.insert(table.clone(), offset_kind_for_data_type(&data_type));
+ }
+ Ok(kinds)
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn serialize_pending_state(&self, state: &State) -> Result<ConnectorState, Error> {
+ self.serialize_state(state).ok_or_else(|| {
+ Error::Serialization(format!("failed to serialize {CONNECTOR_NAME} state"))
+ })
+ }
+
+ async fn retiring_state(&self) -> Option<State> {
+ self.pending_batch
+ .lock()
+ .await
+ .as_ref()
+ .filter(|pending| pending.retiring)
+ .map(|pending| pending.state.clone())
+ }
+
+ fn primary_key_column(&self) -> &str {
+ self.config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(self.tracking_column())
+ }
+
+ fn cleanup_target(&self) -> Option<CleanupTarget> {
+ let action = if self.config.delete_after_read.unwrap_or(false) {
+ CleanupAction::Delete
+ } else if let Some(column) = &self.config.processed_column {
+ CleanupAction::MarkProcessed {
+ column: column.clone(),
+ }
+ } else {
+ return None;
+ };
+ Some(CleanupTarget {
+ key_column: self.primary_key_column().to_string(),
+ action,
+ })
+ }
+
+ /// Restored operations were staged under the config of the run that wrote them.
+ /// Running them under a different key or action would touch rows nobody read.
+ fn validate_pending_cleanup(&mut self) -> Result<(), Error> {
+ let configured = self.cleanup_target();
+ for operation in &self.state.get_mut().pending_operations {
+ if configured.as_ref() != Some(&operation.target) {
+ return Err(Error::InitError(format!(
+ "the cleanup configuration changed while cleanup of {} delivered row(s) in \
+ table '{}' is pending; restore primary_key_column '{}' and action {:?}",
+ operation.ids.len(),
+ operation.table,
+ operation.target.key_column,
+ operation.target.action
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ /// `WHERE key IN (...)` on a key that repeats marks or deletes every row sharing a
+ /// value, including ones that were never read. Tables that don't exist yet are
+ /// checked by the first poll that stages cleanup for them.
+ async fn validate_cleanup_keys(&mut self) -> Result<(), Error> {
+ let Some(target) = self.cleanup_target() else {
+ return Ok(());
+ };
+ let pool = self.get_pool()?;
+ let mut verified = HashSet::with_capacity(self.config.tables.len());
+ for table in &self.config.tables {
+ let check = cleanup_key_check(pool, table, &target.key_column)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to check primary_key_column '{}' on table '{table}': {e}",
+ target.key_column
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ verified.insert(table.clone());
+ }
+ CleanupKeyCheck::NotUnique => {
+ return Err(Error::InitError(not_unique_key_message(
+ table,
+ &target.key_column,
+ )));
+ }
+ CleanupKeyCheck::TableMissing => {}
+ }
+ }
+ *self.verified_cleanup_keys.get_mut() = verified;
+ Ok(())
+ }
+
+ async fn cleanup_key_is_unique(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ key_column: &str,
+ ) -> Result<bool, Error> {
+ if self.verified_cleanup_keys.lock().await.contains(table) {
+ return Ok(true);
+ }
+ let check = cleanup_key_check(pool, table, key_column)
+ .await
+ .map_err(|e| {
+ Error::Connection(format!(
+ "failed to check primary_key_column '{key_column}' on table '{table}': {e}"
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ self.verified_cleanup_keys
+ .lock()
+ .await
+ .insert(table.to_string());
+ Ok(true)
+ }
+ CleanupKeyCheck::NotUnique => Ok(false),
+ CleanupKeyCheck::TableMissing => Err(Error::Connection(format!(
+ "table '{table}' disappeared before its primary_key_column could be checked"
+ ))),
+ }
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ /// Best current knowledge of how a table's tracking values compare.
+ async fn tracking_kind(&self, table: &str) -> OffsetKind {
+ self.tracking_kinds
+ .lock()
+ .await
+ .get(table)
+ .copied()
+ .unwrap_or(OffsetKind::Unknown)
+ }
+
+ /// Whether the stored cursor is what limits the rows a poll can see. The built
+ /// polling query always filters on it; a `custom_query` does so only if it
+ /// interpolates `$offset`.
+ fn cursor_gates_query(&self) -> bool {
+ match &self.config.custom_query {
+ Some(query) => query.contains("$offset"),
+ None => true,
+ }
+ }
+
+ /// The offset a poll actually resumes from: the stored cursor, or the configured
+ /// starting point before anything has been stored.
+ fn effective_offset(&self, last_offset: &Option<String>) -> Option<String> {
+ last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let offset_value = self.effective_offset(last_offset).unwrap_or_default();
+ let offset = format_offset_value(&offset_value, kind);
+ let quoted_table = quote_qualified_identifier(table)?;
+ let limit = batch_size.to_string();
+ let now = Utc::now();
+ let now_unix = now.timestamp().to_string();
+ let now = now.to_rfc3339();
+ let replacements = [
+ ("$table", quoted_table.as_str()),
+ ("$offset", offset.as_str()),
+ ("$limit", limit.as_str()),
+ ("$now_unix", now_unix.as_str()),
+ ("$now", now.as_str()),
+ ];
+
+ Ok(substitute_query_tokens(query, &replacements))
+ }
+
+ /// The query is substituted as text and re-executed once per configured table on
+ /// every poll for the life of the process, so a shape that cannot terminate does
+ /// not fail, it repeats. What can be judged from the text alone is judged here,
+ /// at startup, rather than left to the first poll.
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let tracking_column = self.tracking_column();
+
+ match leading_statement_token(query) {
+ None => {
+ return Err(Error::InitError(
+ "custom_query has no statement: it is empty, or nothing but comments"
+ .to_string(),
+ ));
+ }
+ Some(token) if !READ_STATEMENT_STARTS.contains(&token.as_str()) => {
+ return Err(Error::InitError(format!(
+ "custom_query must be a read starting with SELECT, WITH or TABLE, found \
+ '{token}'"
+ )));
+ }
+ Some(_) => {}
+ }
+
+ // Collapsed so `ORDER\nBY` and `ORDER BY` read the same as `ORDER BY`; the
+ // check hard-fails, so a formatting difference must not decide it.
+ let normalized = query
+ .split_whitespace()
+ .collect::<Vec<_>>()
+ .join(" ")
+ .to_uppercase();
+
+ if query.contains("$offset") {
+ if !normalized.contains("ORDER BY") {
+ return Err(Error::InitError(format!(
+ "custom_query filters on $offset but has no ORDER BY, add \
+ `ORDER BY {tracking_column}`"
+ )));
+ }
+ } else if !self.config.delete_after_read.unwrap_or(false)
+ && self.config.processed_column.is_none()
+ {
+ return Err(Error::InitError(format!(
+ "custom_query must filter with $offset (`WHERE {tracking_column} > $offset`), or \
+ set delete_after_read or processed_column"
+ )));
+ }
+
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "custom_query uses $table but no tables are configured".to_string(),
+ ));
+ }
+ // Without $table the same SQL is executed once per table, so identical rows
+ // would be emitted repeatedly, each carrying a different table's metadata,
+ // message ID, and tracking offset. Table identity is not recoverable from a
+ // static query, so reject the combination instead of guessing one.
+ if !query.contains("$table") && self.config.tables.len() > 1 {
+ return Err(Error::InitError(format!(
+ "custom_query has no $table placeholder but {} tables are configured ({}), use \
+ $table in the query or configure a single table",
+ self.config.tables.len(),
+ self.config.tables.join(", ")
+ )));
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = self.effective_offset(last_offset) {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(&offset, kind)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn apply_pending_operations(
+ &self,
+ operations: Vec<PendingOperation>,
+ deadline: tokio::time::Instant,
+ ) -> Result<Vec<PendingOperation>, Error> {
+ let mut failed_operations = Vec::new();
+ let mut operations = operations.into_iter();
+ while let Some(operation) = operations.next() {
+ if tokio::time::Instant::now() >= deadline {
+ failed_operations.push(operation);
+ failed_operations.extend(operations);
+ break;
+ }
+ let result = tokio::time::timeout_at(deadline, async {
+ let pool = self.get_pool()?;
+ self.mark_or_delete_processed_rows(pool, &operation).await
+ })
+ .await;
+ match result {
+ Ok(Ok(())) => {
+ self.consecutive_cleanup_failures
+ .store(0, Ordering::Relaxed);
+ }
+ Ok(Err(error)) => {
+ self.record_cleanup_failure(&operation, &error)?;
+ failed_operations.push(operation);
+ }
+ Err(_) => {
+ self.record_cleanup_failure(
+ &operation,
+ "cleanup exceeded the 10s budget for one Ack",
+ )?;
+ failed_operations.push(operation);
+ }
+ }
+ }
+ Ok(failed_operations)
+ }
+
+ fn record_cleanup_failure(
+ &self,
+ operation: &PendingOperation,
+ error: impl std::fmt::Display,
+ ) -> Result<(), Error> {
+ let consecutive_failures = self
+ .consecutive_cleanup_failures
+ .fetch_add(1, Ordering::Relaxed)
+ .saturating_add(1);
+ error!(
+ "Failed to mark or delete {} delivered row(s) in table '{}' for {CONNECTOR_NAME} \
+ connector ID: {}, retrying on the next poll ({consecutive_failures} consecutive \
+ failures): {error}",
+ operation.ids.len(),
+ operation.table,
+ self.id
+ );
+ if consecutive_failures >= MAX_CONSECUTIVE_CLEANUP_FAILURES {
+ return Err(Error::Connection(format!(
+ "stopping {CONNECTOR_NAME} connector ID {} after {consecutive_failures} \
+ consecutive mark/delete failures; the cleanup stays in the saved state and \
+ runs again on restart",
+ self.id
+ )));
+ }
+ Ok(())
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ operation: &PendingOperation,
+ ) -> Result<(), Error> {
+ let PendingOperation { table, ids, target } = operation;
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(&target.key_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = match &target.action {
+ CleanupAction::Delete => {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
+ }
+ CleanupAction::MarkProcessed { column } => {
+ let quoted_processed = quote_identifier(column)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({placeholders})"
+ )
+ }
+ };
+
+ // The query and its binds are rebuilt on every attempt because with_retry
+ // calls the closure once per try and each `execute` consumes the query.
+ with_retry(
+ || {
+ let mut prepared = sqlx::query(sqlx::AssertSqlSafe(query.as_str()));
+ for id in ids {
+ prepared = match id {
+ PkValue::Text(text) => prepared.bind(text),
+ PkValue::Bytes(bytes) => prepared.bind(bytes),
+ };
+ }
+ prepared.execute(pool)
+ },
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<PolledBatch, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.tracking_column();
+ let pk_column = self.primary_key_column();
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ //
+ // An ordering violation is handled one step further. Retrying it would
+ // re-run the same query, re-decode the same rows, and hit the same failure
+ // forever, so the table is latched off instead of being retried each cycle.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ let poll_number = self.polls.fetch_add(1, Ordering::Relaxed) + 1;
+ let poisoned = { self.poisoned_tables.lock().await.clone() };
+ let mut newly_poisoned: Vec<String> = Vec::new();
+
+ for table in &self.config.tables {
+ if poisoned.contains(table) {
+ continue;
+ }
+
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ if !batch.messages.is_empty() {
+ self.cursor_regressions.lock().await.remove(table);
+ }
+ // A full batch means the LIMIT was hit, so more rows are waiting.
+ any_batch_full |= batch.messages.len() as u32 >= batch_size;
+ batches.push(batch);
+ }
+ Err(FetchError::Ordering(reason) | FetchError::MissingTrackingColumn(reason)) => {
+ error!(
+ "Table '{table}' at offset {}: {reason}. Table disabled until restart.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ newly_poisoned.push(table.clone());
+ }
+ Err(FetchError::CursorRegression(reason)) => {
+ // Cleared only by a batch that carried rows, so the run counts polls
+ // since the table last published anything. An idle poll produces as
+ // little as a rejected one does and must not read as recovery.
+ let consecutive = {
+ let mut regressions = self.cursor_regressions.lock().await;
+ let consecutive = regressions.entry(table.clone()).or_insert(0);
+ *consecutive += 1;
+ *consecutive
+ };
+ let offset = last_offset.as_deref().unwrap_or("<start>");
+ if consecutive >= CURSOR_REGRESSION_ERROR_THRESHOLD {
+ error!(
+ "Table '{table}' at offset {offset}: {reason}. Skipping this poll \
+ ({consecutive} consecutive polls)."
+ );
+ } else {
+ warn!("Table '{table}' at offset {offset}: {reason}. Skipping this poll.");
+ }
+ }
+ Err(FetchError::StaleOffsetKind(reason)) => {
+ warn!(
+ "Table '{table}' at offset {}: {reason}. Retrying on the next poll.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::UnusableTrackingValue(reason)) => {
+ error!(
+ "Table '{table}' at offset {}: {reason}. Retrying each poll until the row \
+ is fixed.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::Other(error)) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ if !newly_poisoned.is_empty() {
+ let mut poisoned_tables = self.poisoned_tables.lock().await;
+ poisoned_tables.extend(newly_poisoned);
+ if self
+ .config
+ .tables
+ .iter()
+ .all(|table| poisoned_tables.contains(table))
+ {
+ error!(
+ "All configured tables are disabled because their batches cannot yield a \
+ cursor; {CONNECTOR_NAME} connector with ID: {} is idle until restarted",
+ self.id
+ );
+ }
+ }
+
+ // From the next poll on a disabled table is skipped before anything is logged,
+ // so the one line written when it was disabled can scroll out of the operator's
+ // log window and leave a dark connector looking healthy.
+ if poll_number.is_multiple_of(DISABLED_TABLES_LOG_INTERVAL) {
+ let poisoned_tables = self.poisoned_tables.lock().await;
+ if !poisoned_tables.is_empty() {
+ let mut disabled: Vec<&str> = poisoned_tables.iter().map(String::as_str).collect();
+ disabled.sort_unstable();
+ error!(
+ "{} of {} configured table(s) are still disabled and are not being polled by \
+ {CONNECTOR_NAME} connector with ID: {}, and will stay that way until it is \
+ restarted: {}",
+ disabled.len(),
+ self.config.tables.len(),
+ self.id,
+ disabled.join(", ")
+ );
+ }
+ }
+
+ // Phase 2: stage each table's batch. Nothing touches MySQL or the live state
+ // here: the offsets and the mark/delete are applied in `on_batch_result` once
+ // the runtime has sent the batch and saved this candidate state, and a Nack
+ // throws the candidate away so the same rows are read again.
+ let cleanup_target = self.cleanup_target();
+ let mut candidate_state = self.state.lock().await.clone();
+ let mut operations = Vec::new();
+ let mut messages = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if let Some(target) = &cleanup_target
+ && !batch.messages.is_empty()
+ {
+ if batch.processed_ids.is_empty() {
+ error!(
+ "Table '{}': mark/delete is configured but no primary keys were extracted \
+ from {} row(s), skipping this cycle so rows are not published without \
+ being marked or deleted (check that '{pk_column}' is projected and scalar)",
+ batch.table,
+ batch.messages.len()
+ );
+ continue;
+ }
+
+ match self
+ .cleanup_key_is_unique(pool, &batch.table, &target.key_column)
+ .await
+ {
+ Ok(true) => {}
+ Ok(false) => {
+ error!(
+ "{}. Table disabled until restart.",
+ not_unique_key_message(&batch.table, &target.key_column)
+ );
+ self.poisoned_tables.lock().await.insert(batch.table);
+ continue;
+ }
+ Err(error) => {
+ error!("Table '{}' skipped this cycle: {error}", batch.table);
+ continue;
+ }
+ }
+
+ operations.push(PendingOperation {
+ table: batch.table.clone(),
+ ids: std::mem::take(&mut batch.processed_ids),
+ target: target.clone(),
+ });
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ candidate_state.tracking_offsets.insert(batch.table, offset);
+ }
+ }
+
+ let pending = (total_processed > 0).then(|| {
+ candidate_state.processed_rows += total_processed;
+ candidate_state.pending_operations = operations;
+ PendingBatch {
+ state: candidate_state,
+ acknowledged: false,
+ retiring: false,
+ }
+ });
+
+ self.last_batch_full
+ .store(any_batch_full, Ordering::Relaxed);
+ Ok(PolledBatch { messages, pending })
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, FetchError> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query_kind = self.tracking_kind(table).await;
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.substitute_query_params(custom_query, table, last_offset, batch_size, query_kind)?
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size, query_kind)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
Review Comment:
warning: custom queries without `LIMIT` bypass `batch_size`, so reads can use unbounded memory and exceed cleanup's placeholder limit. enforce `batch_size` before materializing rows or staging cleanup.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,4071 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source,
+ source::SourceBatchResult, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+/// Upper bound on applying a batch's staged mark/delete after its Ack. The SDK waits
+/// `BATCH_RESULT_TIMEOUT` (30s) for the result, so this has to finish well inside it.
+const ACK_BATCH_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// MySQL rejects a prepared statement with more placeholders than this (error 1390), and
+/// the mark/delete binds one per row of the batch.
+const MAX_CLEANUP_BATCH_SIZE: u32 = 65_535;
+
+/// How many mark/delete statements in a row may fail before the connector stops. The
+/// operations stay in the persisted state and run again on restart.
+const MAX_CONSECUTIVE_CLEANUP_FAILURES: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ pending_batch: Mutex<Option<PendingBatch>>,
+ consecutive_cleanup_failures: AtomicU32,
+ verified_cleanup_keys: Mutex<HashSet<String>>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some("it has too few distinct values to be unique per row");
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some("binary values are compared as base64 text, which does not keep byte order");
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some("MySQL orders these by declaration index, not by label");
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL \
+ or not an ordered scalar (tinyint(1), JSON)"
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "row '{value}' is below the stored offset '{previous}' for tracking_column \
+ '{column}'. Filter the query with `{column} > $offset`"
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. Order the query ascending by '{column}'"
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+ /// Mark/delete work for a batch the runtime has already sent. It is persisted with
+ /// the batch's offsets so a crash between the send and the cleanup reruns it.
+ #[serde(default)]
+ pending_operations: Vec<PendingOperation>,
+}
+
+/// The rows of one table to mark or delete once their batch is acknowledged.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct PendingOperation {
+ table: String,
+ ids: Vec<PkValue>,
+ target: CleanupTarget,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct CleanupTarget {
+ key_column: String,
+ action: CleanupAction,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum CleanupAction {
+ Delete,
+ MarkProcessed { column: String },
+}
+
+/// The candidate state of the batch most recently handed to the runtime. Live state
+/// only takes it over on Ack, so a Nack leaves offsets and rows where they were.
+#[derive(Debug, Clone)]
+struct PendingBatch {
+ state: State,
+ /// The runtime has delivered the batch but its cleanup has not finished yet.
+ acknowledged: bool,
+ /// Cleanup finished and the next poll must persist `state` without the operations,
+ /// so a restart does not run them again.
+ retiring: bool,
+}
+
+struct PolledBatch {
+ messages: Vec<ProducedMessage>,
+ pending: Option<PendingBatch>,
+}
+
+enum CleanupKeyCheck {
+ Unique,
+ NotUnique,
+ TableMissing,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty, list the table even when custom_query hardcodes it"
+ .to_string(),
+ ));
+ }
+
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ if let Some(batch_size) = self.config.batch_size
+ && batch_size > MAX_CLEANUP_BATCH_SIZE
+ && self.cleanup_target().is_some()
+ {
+ return Err(Error::InitError(format!(
+ "batch_size {batch_size} exceeds {MAX_CLEANUP_BATCH_SIZE} with delete_after_read or \
+ processed_column set, MySQL allows at most {MAX_CLEANUP_BATCH_SIZE} placeholders \
+ in the statement that marks or deletes a batch"
+ )));
+ }
+
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "name the column custom_query orders by and filters with $offset"
+ } else {
+ "use a unique, monotonically increasing column"
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set, {guidance}"
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.validate_pending_cleanup()?;
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+ self.validate_cleanup_keys().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ if let Some(state) = self.retiring_state().await {
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: Some(self.serialize_pending_state(&state)?),
+ });
+ }
+
+ // Skip the pacing delay while draining a backlog, but never while a cleanup
+ // retry is pending, or a failing mark/delete would spin without a pause.
+ let cleanup_pending = self.pending_batch.lock().await.is_some();
+ if cleanup_pending || !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ {
+ let pending = self.pending_batch.lock().await;
+ if let Some(pending) = pending.as_ref() {
+ if !pending.acknowledged {
+ error!(
+ "{CONNECTOR_NAME} connector ID: {} was polled while a batch was still in flight",
+ self.id
+ );
+ return Err(Error::InvalidState);
+ }
+ // An empty batch whose Ack retries the cleanup the last one left behind.
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: None,
+ });
+ }
+ }
+
+ let polled = self.poll_tables().await?;
+
+ let processed_rows = match &polled.pending {
+ Some(pending) => pending.state.processed_rows,
+ None => self.state.lock().await.processed_rows,
+ };
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ }
+
+ // Idle cycles stage nothing, so return None to let the runtime skip the fsync.
+ let persisted_state = polled
+ .pending
+ .as_ref()
+ .map(|pending| self.serialize_pending_state(&pending.state))
+ .transpose()?;
+ *self.pending_batch.lock().await = polled.pending;
+
+ Ok(ProducedMessages {
+ schema,
+ messages: polled.messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> {
+ if result == SourceBatchResult::Nack {
+ // Dropping the candidate leaves the live offsets where they were, so the
+ // next poll reads the same rows again. Delivered batches keep their cleanup.
+ let mut pending = self.pending_batch.lock().await;
+ if pending
+ .as_ref()
+ .is_some_and(|pending| !pending.acknowledged && !pending.retiring)
+ {
+ pending.take();
+ }
+ return Ok(());
+ }
+
+ let Some(PendingBatch { mut state, .. }) = self.pending_batch.lock().await.clone() else {
+ return Ok(());
+ };
+
+ let operations = std::mem::take(&mut state.pending_operations);
+ let had_operations = !operations.is_empty();
+ let deadline = tokio::time::Instant::now() + ACK_BATCH_TIMEOUT;
+ let failed_operations = self.apply_pending_operations(operations, deadline).await?;
+
+ if failed_operations.is_empty() {
+ *self.state.lock().await = state.clone();
+ let mut pending = self.pending_batch.lock().await;
+ if had_operations {
+ *pending = Some(PendingBatch {
+ state,
+ acknowledged: false,
+ retiring: true,
+ });
+ } else {
+ pending.take();
+ }
+ } else {
+ state.pending_operations = failed_operations;
+ *self.pending_batch.lock().await = Some(PendingBatch {
+ state,
+ acknowledged: true,
+ retiring: false,
+ });
+ }
+
+ Ok(())
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ let state = restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ pending_operations: Vec::new(),
+ });
+ // The runtime only persists a batch's operations after sending it, so restored
+ // operations belong to a delivered batch and only the cleanup is left to do.
+ let pending_batch = (!state.pending_operations.is_empty()).then(|| PendingBatch {
+ state: state.clone(),
+ acknowledged: true,
+ retiring: false,
+ });
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(state),
+ pending_batch: Mutex::new(pending_batch),
+ consecutive_cleanup_failures: AtomicU32::new(0),
+ verified_cleanup_keys: Mutex::new(HashSet::new()),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ poisoned_tables: Mutex::new(HashSet::new()),
+ tracking_kinds: Mutex::new(HashMap::new()),
+ polls: AtomicU64::new(0),
+ cursor_regressions: Mutex::new(HashMap::new()),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ // `format_offset_value` escapes backslashes, which under NO_BACKSLASH_ESCAPES
+ // would reach MySQL as literal characters and filter on the wrong cursor.
+ .after_connect(|connection, _| {
+ Box::pin(async move {
+ sqlx::query(
+ "SET SESSION sql_mode = REPLACE(@@SESSION.sql_mode, 'NO_BACKSLASH_ESCAPES', '')",
+ )
+ .execute(&mut *connection)
+ .await?;
+ Ok(())
+ })
+ })
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn tracking_column(&self) -> &str {
+ self.config.tracking_column.as_deref().unwrap_or("id")
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn resolve_tracking_columns(&self) -> Result<HashMap<String, OffsetKind>, Error> {
+ // Whether the connector owns the query, and can therefore hold the schema to
+ // what that query will do with it.
+ let strict = self.config.custom_query.is_none();
+ let pool = self.get_pool()?;
+ let tracking_column = self.tracking_column();
+ let mut kinds = HashMap::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ }
+
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ }
+
+ warn!(
+ "tracking_column '{tracking_column}' is not a column of table '{table}'; \
+ assuming custom_query projects it. Its comparison order is resolved from \
+ the first result set instead."
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ // The column's type survives a plain projection, so it is worth rejecting
+ // whatever the query looks like. A custom_query aliasing an expression onto
+ // a name the table also uses is the one false positive, hence the hint.
+ if let Some(reason) = unusable_tracking_type(&data_type) {
+ let aliasing_hint = if self.config.custom_query.is_some() {
+ " If custom_query aliases a different expression to this name, rename the \
+ alias so it does not collide with the column."
+ } else {
+ ""
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': {reason}. Use an integer, timestamp, or string \
+ column.{aliasing_hint}"
+ )));
+ }
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; a row \
+ whose value is NULL cannot produce a cursor and will fail that table's batch. \
+ Have custom_query exclude those rows, or declare the column NOT NULL."
+ );
+ }
+
+ if is_collation_ordered(&data_type) {
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': values must be unique and increase in its collation order \
+ (zero-pad numbers, prefer a _bin collation)"
+ );
+ }
+
+ kinds.insert(table.clone(), offset_kind_for_data_type(&data_type));
+ }
+ Ok(kinds)
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn serialize_pending_state(&self, state: &State) -> Result<ConnectorState, Error> {
+ self.serialize_state(state).ok_or_else(|| {
+ Error::Serialization(format!("failed to serialize {CONNECTOR_NAME} state"))
+ })
+ }
+
+ async fn retiring_state(&self) -> Option<State> {
+ self.pending_batch
+ .lock()
+ .await
+ .as_ref()
+ .filter(|pending| pending.retiring)
+ .map(|pending| pending.state.clone())
+ }
+
+ fn primary_key_column(&self) -> &str {
+ self.config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(self.tracking_column())
+ }
+
+ fn cleanup_target(&self) -> Option<CleanupTarget> {
+ let action = if self.config.delete_after_read.unwrap_or(false) {
+ CleanupAction::Delete
+ } else if let Some(column) = &self.config.processed_column {
+ CleanupAction::MarkProcessed {
+ column: column.clone(),
+ }
+ } else {
+ return None;
+ };
+ Some(CleanupTarget {
+ key_column: self.primary_key_column().to_string(),
+ action,
+ })
+ }
+
+ /// Restored operations were staged under the config of the run that wrote them.
+ /// Running them under a different key or action would touch rows nobody read.
+ fn validate_pending_cleanup(&mut self) -> Result<(), Error> {
+ let configured = self.cleanup_target();
+ for operation in &self.state.get_mut().pending_operations {
+ if configured.as_ref() != Some(&operation.target) {
+ return Err(Error::InitError(format!(
+ "the cleanup configuration changed while cleanup of {} delivered row(s) in \
+ table '{}' is pending; restore primary_key_column '{}' and action {:?}",
+ operation.ids.len(),
+ operation.table,
+ operation.target.key_column,
+ operation.target.action
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ /// `WHERE key IN (...)` on a key that repeats marks or deletes every row sharing a
+ /// value, including ones that were never read. Tables that don't exist yet are
+ /// checked by the first poll that stages cleanup for them.
+ async fn validate_cleanup_keys(&mut self) -> Result<(), Error> {
+ let Some(target) = self.cleanup_target() else {
+ return Ok(());
+ };
+ let pool = self.get_pool()?;
+ let mut verified = HashSet::with_capacity(self.config.tables.len());
+ for table in &self.config.tables {
+ let check = cleanup_key_check(pool, table, &target.key_column)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to check primary_key_column '{}' on table '{table}': {e}",
+ target.key_column
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ verified.insert(table.clone());
+ }
+ CleanupKeyCheck::NotUnique => {
+ return Err(Error::InitError(not_unique_key_message(
+ table,
+ &target.key_column,
+ )));
+ }
+ CleanupKeyCheck::TableMissing => {}
+ }
+ }
+ *self.verified_cleanup_keys.get_mut() = verified;
+ Ok(())
+ }
+
+ async fn cleanup_key_is_unique(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ key_column: &str,
+ ) -> Result<bool, Error> {
+ if self.verified_cleanup_keys.lock().await.contains(table) {
+ return Ok(true);
+ }
+ let check = cleanup_key_check(pool, table, key_column)
+ .await
+ .map_err(|e| {
+ Error::Connection(format!(
+ "failed to check primary_key_column '{key_column}' on table '{table}': {e}"
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ self.verified_cleanup_keys
+ .lock()
+ .await
+ .insert(table.to_string());
+ Ok(true)
+ }
+ CleanupKeyCheck::NotUnique => Ok(false),
+ CleanupKeyCheck::TableMissing => Err(Error::Connection(format!(
+ "table '{table}' disappeared before its primary_key_column could be checked"
+ ))),
+ }
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ /// Best current knowledge of how a table's tracking values compare.
+ async fn tracking_kind(&self, table: &str) -> OffsetKind {
+ self.tracking_kinds
+ .lock()
+ .await
+ .get(table)
+ .copied()
+ .unwrap_or(OffsetKind::Unknown)
+ }
+
+ /// Whether the stored cursor is what limits the rows a poll can see. The built
+ /// polling query always filters on it; a `custom_query` does so only if it
+ /// interpolates `$offset`.
+ fn cursor_gates_query(&self) -> bool {
+ match &self.config.custom_query {
+ Some(query) => query.contains("$offset"),
+ None => true,
+ }
+ }
+
+ /// The offset a poll actually resumes from: the stored cursor, or the configured
+ /// starting point before anything has been stored.
+ fn effective_offset(&self, last_offset: &Option<String>) -> Option<String> {
+ last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let offset_value = self.effective_offset(last_offset).unwrap_or_default();
+ let offset = format_offset_value(&offset_value, kind);
+ let quoted_table = quote_qualified_identifier(table)?;
+ let limit = batch_size.to_string();
+ let now = Utc::now();
+ let now_unix = now.timestamp().to_string();
+ let now = now.to_rfc3339();
+ let replacements = [
+ ("$table", quoted_table.as_str()),
+ ("$offset", offset.as_str()),
+ ("$limit", limit.as_str()),
+ ("$now_unix", now_unix.as_str()),
+ ("$now", now.as_str()),
+ ];
+
+ Ok(substitute_query_tokens(query, &replacements))
+ }
+
+ /// The query is substituted as text and re-executed once per configured table on
+ /// every poll for the life of the process, so a shape that cannot terminate does
+ /// not fail, it repeats. What can be judged from the text alone is judged here,
+ /// at startup, rather than left to the first poll.
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let tracking_column = self.tracking_column();
+
+ match leading_statement_token(query) {
+ None => {
+ return Err(Error::InitError(
+ "custom_query has no statement: it is empty, or nothing but comments"
+ .to_string(),
+ ));
+ }
+ Some(token) if !READ_STATEMENT_STARTS.contains(&token.as_str()) => {
+ return Err(Error::InitError(format!(
+ "custom_query must be a read starting with SELECT, WITH or TABLE, found \
+ '{token}'"
+ )));
+ }
+ Some(_) => {}
+ }
+
+ // Collapsed so `ORDER\nBY` and `ORDER BY` read the same as `ORDER BY`; the
+ // check hard-fails, so a formatting difference must not decide it.
+ let normalized = query
+ .split_whitespace()
+ .collect::<Vec<_>>()
+ .join(" ")
+ .to_uppercase();
+
+ if query.contains("$offset") {
+ if !normalized.contains("ORDER BY") {
+ return Err(Error::InitError(format!(
+ "custom_query filters on $offset but has no ORDER BY, add \
+ `ORDER BY {tracking_column}`"
+ )));
+ }
+ } else if !self.config.delete_after_read.unwrap_or(false)
+ && self.config.processed_column.is_none()
+ {
+ return Err(Error::InitError(format!(
+ "custom_query must filter with $offset (`WHERE {tracking_column} > $offset`), or \
+ set delete_after_read or processed_column"
+ )));
+ }
+
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "custom_query uses $table but no tables are configured".to_string(),
+ ));
+ }
+ // Without $table the same SQL is executed once per table, so identical rows
+ // would be emitted repeatedly, each carrying a different table's metadata,
+ // message ID, and tracking offset. Table identity is not recoverable from a
+ // static query, so reject the combination instead of guessing one.
+ if !query.contains("$table") && self.config.tables.len() > 1 {
+ return Err(Error::InitError(format!(
+ "custom_query has no $table placeholder but {} tables are configured ({}), use \
+ $table in the query or configure a single table",
+ self.config.tables.len(),
+ self.config.tables.join(", ")
+ )));
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = self.effective_offset(last_offset) {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(&offset, kind)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn apply_pending_operations(
+ &self,
+ operations: Vec<PendingOperation>,
+ deadline: tokio::time::Instant,
+ ) -> Result<Vec<PendingOperation>, Error> {
+ let mut failed_operations = Vec::new();
+ let mut operations = operations.into_iter();
+ while let Some(operation) = operations.next() {
+ if tokio::time::Instant::now() >= deadline {
+ failed_operations.push(operation);
+ failed_operations.extend(operations);
+ break;
+ }
+ let result = tokio::time::timeout_at(deadline, async {
+ let pool = self.get_pool()?;
+ self.mark_or_delete_processed_rows(pool, &operation).await
+ })
+ .await;
+ match result {
+ Ok(Ok(())) => {
+ self.consecutive_cleanup_failures
+ .store(0, Ordering::Relaxed);
+ }
+ Ok(Err(error)) => {
+ self.record_cleanup_failure(&operation, &error)?;
+ failed_operations.push(operation);
+ }
+ Err(_) => {
+ self.record_cleanup_failure(
+ &operation,
+ "cleanup exceeded the 10s budget for one Ack",
+ )?;
+ failed_operations.push(operation);
+ }
+ }
+ }
+ Ok(failed_operations)
+ }
+
+ fn record_cleanup_failure(
+ &self,
+ operation: &PendingOperation,
+ error: impl std::fmt::Display,
+ ) -> Result<(), Error> {
+ let consecutive_failures = self
+ .consecutive_cleanup_failures
+ .fetch_add(1, Ordering::Relaxed)
+ .saturating_add(1);
+ error!(
+ "Failed to mark or delete {} delivered row(s) in table '{}' for {CONNECTOR_NAME} \
+ connector ID: {}, retrying on the next poll ({consecutive_failures} consecutive \
+ failures): {error}",
+ operation.ids.len(),
+ operation.table,
+ self.id
+ );
+ if consecutive_failures >= MAX_CONSECUTIVE_CLEANUP_FAILURES {
+ return Err(Error::Connection(format!(
+ "stopping {CONNECTOR_NAME} connector ID {} after {consecutive_failures} \
+ consecutive mark/delete failures; the cleanup stays in the saved state and \
+ runs again on restart",
+ self.id
+ )));
+ }
+ Ok(())
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ operation: &PendingOperation,
+ ) -> Result<(), Error> {
+ let PendingOperation { table, ids, target } = operation;
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(&target.key_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = match &target.action {
+ CleanupAction::Delete => {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
+ }
+ CleanupAction::MarkProcessed { column } => {
+ let quoted_processed = quote_identifier(column)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({placeholders})"
+ )
+ }
+ };
+
+ // The query and its binds are rebuilt on every attempt because with_retry
+ // calls the closure once per try and each `execute` consumes the query.
+ with_retry(
+ || {
+ let mut prepared = sqlx::query(sqlx::AssertSqlSafe(query.as_str()));
+ for id in ids {
+ prepared = match id {
+ PkValue::Text(text) => prepared.bind(text),
+ PkValue::Bytes(bytes) => prepared.bind(bytes),
+ };
+ }
+ prepared.execute(pool)
+ },
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<PolledBatch, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.tracking_column();
+ let pk_column = self.primary_key_column();
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ //
+ // An ordering violation is handled one step further. Retrying it would
+ // re-run the same query, re-decode the same rows, and hit the same failure
+ // forever, so the table is latched off instead of being retried each cycle.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ let poll_number = self.polls.fetch_add(1, Ordering::Relaxed) + 1;
+ let poisoned = { self.poisoned_tables.lock().await.clone() };
+ let mut newly_poisoned: Vec<String> = Vec::new();
+
+ for table in &self.config.tables {
+ if poisoned.contains(table) {
+ continue;
+ }
+
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ if !batch.messages.is_empty() {
+ self.cursor_regressions.lock().await.remove(table);
+ }
+ // A full batch means the LIMIT was hit, so more rows are waiting.
+ any_batch_full |= batch.messages.len() as u32 >= batch_size;
+ batches.push(batch);
+ }
+ Err(FetchError::Ordering(reason) | FetchError::MissingTrackingColumn(reason)) => {
+ error!(
+ "Table '{table}' at offset {}: {reason}. Table disabled until restart.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ newly_poisoned.push(table.clone());
+ }
+ Err(FetchError::CursorRegression(reason)) => {
+ // Cleared only by a batch that carried rows, so the run counts polls
+ // since the table last published anything. An idle poll produces as
+ // little as a rejected one does and must not read as recovery.
+ let consecutive = {
+ let mut regressions = self.cursor_regressions.lock().await;
+ let consecutive = regressions.entry(table.clone()).or_insert(0);
+ *consecutive += 1;
+ *consecutive
+ };
+ let offset = last_offset.as_deref().unwrap_or("<start>");
+ if consecutive >= CURSOR_REGRESSION_ERROR_THRESHOLD {
+ error!(
+ "Table '{table}' at offset {offset}: {reason}. Skipping this poll \
+ ({consecutive} consecutive polls)."
+ );
+ } else {
+ warn!("Table '{table}' at offset {offset}: {reason}. Skipping this poll.");
+ }
+ }
+ Err(FetchError::StaleOffsetKind(reason)) => {
+ warn!(
+ "Table '{table}' at offset {}: {reason}. Retrying on the next poll.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::UnusableTrackingValue(reason)) => {
+ error!(
+ "Table '{table}' at offset {}: {reason}. Retrying each poll until the row \
+ is fixed.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::Other(error)) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ if !newly_poisoned.is_empty() {
+ let mut poisoned_tables = self.poisoned_tables.lock().await;
+ poisoned_tables.extend(newly_poisoned);
+ if self
+ .config
+ .tables
+ .iter()
+ .all(|table| poisoned_tables.contains(table))
+ {
+ error!(
+ "All configured tables are disabled because their batches cannot yield a \
+ cursor; {CONNECTOR_NAME} connector with ID: {} is idle until restarted",
+ self.id
+ );
+ }
+ }
+
+ // From the next poll on a disabled table is skipped before anything is logged,
+ // so the one line written when it was disabled can scroll out of the operator's
+ // log window and leave a dark connector looking healthy.
+ if poll_number.is_multiple_of(DISABLED_TABLES_LOG_INTERVAL) {
+ let poisoned_tables = self.poisoned_tables.lock().await;
+ if !poisoned_tables.is_empty() {
+ let mut disabled: Vec<&str> = poisoned_tables.iter().map(String::as_str).collect();
+ disabled.sort_unstable();
+ error!(
+ "{} of {} configured table(s) are still disabled and are not being polled by \
+ {CONNECTOR_NAME} connector with ID: {}, and will stay that way until it is \
+ restarted: {}",
+ disabled.len(),
+ self.config.tables.len(),
+ self.id,
+ disabled.join(", ")
+ );
+ }
+ }
+
+ // Phase 2: stage each table's batch. Nothing touches MySQL or the live state
+ // here: the offsets and the mark/delete are applied in `on_batch_result` once
+ // the runtime has sent the batch and saved this candidate state, and a Nack
+ // throws the candidate away so the same rows are read again.
+ let cleanup_target = self.cleanup_target();
+ let mut candidate_state = self.state.lock().await.clone();
+ let mut operations = Vec::new();
+ let mut messages = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if let Some(target) = &cleanup_target
+ && !batch.messages.is_empty()
+ {
+ if batch.processed_ids.is_empty() {
+ error!(
+ "Table '{}': mark/delete is configured but no primary keys were extracted \
+ from {} row(s), skipping this cycle so rows are not published without \
+ being marked or deleted (check that '{pk_column}' is projected and scalar)",
+ batch.table,
+ batch.messages.len()
+ );
+ continue;
+ }
+
+ match self
+ .cleanup_key_is_unique(pool, &batch.table, &target.key_column)
+ .await
+ {
+ Ok(true) => {}
+ Ok(false) => {
+ error!(
+ "{}. Table disabled until restart.",
+ not_unique_key_message(&batch.table, &target.key_column)
+ );
+ self.poisoned_tables.lock().await.insert(batch.table);
+ continue;
+ }
+ Err(error) => {
+ error!("Table '{}' skipped this cycle: {error}", batch.table);
+ continue;
+ }
+ }
+
+ operations.push(PendingOperation {
+ table: batch.table.clone(),
+ ids: std::mem::take(&mut batch.processed_ids),
+ target: target.clone(),
+ });
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ candidate_state.tracking_offsets.insert(batch.table, offset);
+ }
+ }
+
+ let pending = (total_processed > 0).then(|| {
+ candidate_state.processed_rows += total_processed;
+ candidate_state.pending_operations = operations;
+ PendingBatch {
+ state: candidate_state,
+ acknowledged: false,
+ retiring: false,
+ }
+ });
+
+ self.last_batch_full
+ .store(any_batch_full, Ordering::Relaxed);
+ Ok(PolledBatch { messages, pending })
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, FetchError> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query_kind = self.tracking_kind(table).await;
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.substitute_query_params(custom_query, table, last_offset, batch_size, query_kind)?
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size, query_kind)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ // Settled once for the whole batch rather than per row, since the column list
+ // is a property of the result set.
+ if let Some(first) = rows.first() {
+ ensure_tracking_column_projected(
+ first.columns().iter().map(|column| column.name()),
+ tracking_column,
+ )?;
+ }
+
+ let resume_from = if self.cursor_gates_query() {
+ self.effective_offset(last_offset)
+ } else {
+ None
+ };
+
+ // The result set describes what this query actually returned, so it outranks
+ // whatever `information_schema` said at startup and is the only answer
+ // available for a computed column.
+ let kind = match rows.first().and_then(|row| {
+ observed_tracking_kind(row, tracking_column).filter(|observed| *observed != query_kind)
+ }) {
+ Some(observed) => {
+ self.tracking_kinds
+ .lock()
+ .await
+ .insert(table.to_string(), observed);
+ // The query was already sent with the stale kind, so its offset literal
+ // may be quoted for the wrong type.
+ if let Some(offset) = &resume_from
+ && offset_literal_diverged(offset, query_kind, observed)
+ {
+ return Err(FetchError::StaleOffsetKind(format!(
+ "tracking_column '{tracking_column}' type differs from the one detected \
+ at startup (filtered with {}, expected {}), batch discarded",
+ format_offset_value(offset, query_kind),
+ format_offset_value(offset, observed)
+ )));
+ }
+ observed
+ }
+ None => query_kind,
+ };
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ // Every row of a result set shares its columns, so the published key names are
+ // derived once per batch instead of once per row.
+ let column_names: Vec<String> = rows
+ .first()
+ .map(|row| {
+ row.columns()
+ .iter()
+ .map(|column| {
+ if table_config.snake_case_columns {
+ to_snake_case(column.name())
+ } else {
+ column.name().to_string()
+ }
+ })
+ .collect()
+ })
+ .unwrap_or_default();
+ let mut ordering = OrderingGuard::new(tracking_column, kind, resume_from);
+ for row in rows {
+ let processed = self
+ .process_row(&row, &table_config, &column_names)
+ .map_err(|e| {
+ error!(
+ "Failed to decode row in table '{table}' (pk {}): {e}",
+ pk_for_log(&row, table_config.pk_column)
+ );
+ e
+ })?;
+
+ let Some(value) = processed.tracking_value else {
+ return Err(unusable_tracking_value(
+ tracking_column,
+ &pk_for_log(&row, table_config.pk_column),
+ ));
+ };
+ ordering
+ .accept(value)
+ .map_err(|violation| match violation {
+ OrderingViolation::WithinBatch(reason) => FetchError::Ordering(reason),
+ OrderingViolation::BelowCursor(reason) => FetchError::CursorRegression(reason),
+ })?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+
+ batch.messages.push(processed.message);
+ }
+ batch.max_offset = ordering.into_cursor();
+
+ Ok(batch)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ column_names: &[String],
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<PkValue> = None;
+ let mut tracking_value: Option<String> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
+ let mut extracted_payload: Option<Vec<u8>> = None;
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ if name.eq_ignore_ascii_case(config.payload_col) {
+ extracted_payload =
+ Some(self.extract_payload_column(row, i, config.payload_format)?);
+ }
+ let is_tracking = name.eq_ignore_ascii_case(config.tracking_column);
+ let is_pk = name.eq_ignore_ascii_case(config.pk_column);
+ if is_tracking {
+ let value = extract_column_value(row, i)?;
+ tracking_value = column_value_as_string(row, i, &value)?;
+ if is_pk {
+ row_pk = pk_value_from_decoded(row, i, &value)?;
+ }
+ } else if is_pk {
+ row_pk = extract_pk_value(row, i)?;
+ }
+ }
+
+ // Every extract_payload_column arm yields Ok for a NULL column, so None
+ // here means the column was absent. Falling back to the whole-row JSON
+ // would publish bytes contradicting the schema derived from payload_format.
+ let payload = extracted_payload.ok_or_else(|| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': payload_column '{}' not present in the result set",
+ config.table, config.payload_col
+ ))
+ })?;
+
+ return Ok(build_processed_row(
+ config.table,
+ payload,
+ tracking_value,
+ row_pk,
+ ));
+ }
+
+ let mut data = serde_json::Map::new();
+ for ((i, column), column_name) in row.columns().iter().enumerate().zip(column_names) {
+ let name = column.name();
+ let value = extract_column_value(row, i)?;
+ if name.eq_ignore_ascii_case(config.tracking_column) {
+ tracking_value = column_value_as_string(row, i, &value)?;
+ }
+ if name.eq_ignore_ascii_case(config.pk_column) {
+ row_pk = pk_value_from_decoded(row, i, &value)?;
+ }
+ data.insert(column_name.clone(), value);
+ }
+
+ let payload = if config.include_metadata {
+ let record = DatabaseRecord {
+ table_name: config.table.to_string(),
+ operation_type: "SELECT".to_string(),
+ timestamp: Utc::now(),
+ data: serde_json::Value::Object(data),
+ old_data: None,
+ };
+ simd_json::to_vec(&record).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': failed to serialize row to JSON: {e}",
+ config.table
+ ))
+ })?
+ } else {
+ simd_json::to_vec(&data).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': failed to serialize row to JSON: {e}",
+ config.table
+ ))
+ })?
+ };
+
+ Ok(build_processed_row(
+ config.table,
+ payload,
+ tracking_value,
+ row_pk,
+ ))
+ }
+}
+
+/// Type-faithful primary-key extraction. Binary columns are taken as raw bytes
+/// rather than routed through the base64 path in `extract_column_value`, so the
+/// value can be bound back into `WHERE pk IN (...)` and match the row. Everything
+/// else takes the string form MySQL parses back to the same value. Returns None
+/// only for a NULL pk, which a real primary key cannot be.
+fn extract_pk_value(row: &MySqlRow, column_index: usize) -> Result<Option<PkValue>, Error> {
+ if is_binary_column(row, column_index) {
+ return binary_pk_value(row, column_index);
+ }
+ let value = extract_column_value(row, column_index)?;
+ pk_value_from_decoded(row, column_index, &value)
+}
+
+/// Like `extract_pk_value` for a column the caller has already decoded, so the
+/// common key column is not decoded a second time per row.
+fn pk_value_from_decoded(
+ row: &MySqlRow,
+ column_index: usize,
+ value: &serde_json::Value,
+) -> Result<Option<PkValue>, Error> {
+ if is_binary_column(row, column_index) {
+ return binary_pk_value(row, column_index);
+ }
+ Ok(column_value_as_string(row, column_index, value)?.map(PkValue::Text))
+}
+
+fn is_binary_column(row: &MySqlRow, column_index: usize) -> bool {
+ matches!(
+ row.columns()[column_index].type_info().name(),
+ "BINARY" | "VARBINARY" | "TINYBLOB" | "BLOB" | "MEDIUMBLOB" | "LONGBLOB"
+ )
+}
+
+fn binary_pk_value(row: &MySqlRow, column_index: usize) -> Result<Option<PkValue>, Error> {
+ let column = &row.columns()[column_index];
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "primary key column '{}' (MySQL type '{}'): {e}",
+ column.name(),
+ column.type_info().name()
+ ))
+ })?;
+ Ok(bytes.map(PkValue::Bytes))
+}
+
+fn extract_column_value(row: &MySqlRow, column_index: usize) -> Result<serde_json::Value, Error> {
+ let column = &row.columns()[column_index];
+ let type_name = column.type_info().name();
+
+ let to_err = |e: sqlx::Error| {
+ Error::InvalidRecordValue(format!(
+ "column '{}' (MySQL type '{type_name}'): {e}",
+ column.name()
+ ))
+ };
+
+ match type_name {
+ "BOOLEAN" => {
+ let value: Option<bool> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::Bool)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT" => {
+ let value: Option<i8> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT" => {
+ let value: Option<i16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT" | "INT" => {
+ let value: Option<i32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT" => {
+ let value: Option<i64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT UNSIGNED" => {
+ let value: Option<u8> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT UNSIGNED" => {
+ let value: Option<u16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT UNSIGNED" | "INT UNSIGNED" => {
+ let value: Option<u32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT UNSIGNED" => {
+ let value: Option<u64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "FLOAT" => {
+ let value: Option<f32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as f64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DOUBLE" => {
+ let value: Option<f64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DECIMAL" => {
+ let value: Option<rust_decimal::Decimal> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|d| serde_json::Value::String(d.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIT" => {
+ let value: Option<u64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "YEAR" => {
+ let value: Option<u16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATE" => {
+ let value: Option<NaiveDate> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|d| serde_json::Value::String(d.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TIME" => {
+ let value: Option<NaiveTime> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|t| serde_json::Value::String(t.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATETIME" => {
+ let value: Option<NaiveDateTime> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|dt| serde_json::Value::String(dt.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TIMESTAMP" => {
+ let value: Option<DateTime<Utc>> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|dt| serde_json::Value::String(dt.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT" | "MEDIUMTEXT" | "LONGTEXT" | "ENUM" | "SET" => {
+ let value: Option<String> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BINARY" | "VARBINARY" | "TINYBLOB" | "BLOB" | "MEDIUMBLOB" | "LONGBLOB" => {
+ let value: Option<Vec<u8>> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|bytes| {
+ serde_json::Value::String(
+ base64::engine::general_purpose::STANDARD.encode(&bytes),
+ )
+ })
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "JSON" => {
+ let value: Option<serde_json::Value> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value.unwrap_or(serde_json::Value::Null))
+ }
+ "NULL" => Ok(serde_json::Value::Null),
+ _ => {
+ let column_name = column.name();
+ warn!(
+ "Column '{column_name}' has unrecognized MySQL type '{type_name}', \
+ attempting text extraction"
+ );
+ if let Ok(text) = row.try_get::<Option<String>, _>(column_index) {
+ return Ok(text
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null));
+ }
+ if let Ok(bytes) = row.try_get::<Option<Vec<u8>>, _>(column_index) {
+ return Ok(bytes
+ .map(|b| {
+ serde_json::Value::String(
+ base64::engine::general_purpose::STANDARD.encode(&b),
+ )
+ })
+ .unwrap_or(serde_json::Value::Null));
+ }
+ error!(
+ "Column '{column_name}' has unsupported MySQL type '{type_name}', \
+ returning null"
+ );
+ Ok(serde_json::Value::Null)
Review Comment:
critical: non-null `GEOMETRY` fields become JSON null when both fallback decoders fail, and the row still advances the cursor. return `Error::InvalidRecordValue` instead of accepting the row.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r4306856998
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,4071 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source,
+ source::SourceBatchResult, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+/// Upper bound on applying a batch's staged mark/delete after its Ack. The SDK waits
+/// `BATCH_RESULT_TIMEOUT` (30s) for the result, so this has to finish well inside it.
+const ACK_BATCH_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// MySQL rejects a prepared statement with more placeholders than this (error 1390), and
+/// the mark/delete binds one per row of the batch.
+const MAX_CLEANUP_BATCH_SIZE: u32 = 65_535;
+
+/// How many mark/delete statements in a row may fail before the connector stops. The
+/// operations stay in the persisted state and run again on restart.
+const MAX_CONSECUTIVE_CLEANUP_FAILURES: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ pending_batch: Mutex<Option<PendingBatch>>,
+ consecutive_cleanup_failures: AtomicU32,
+ verified_cleanup_keys: Mutex<HashSet<String>>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some("it has too few distinct values to be unique per row");
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some("binary values are compared as base64 text, which does not keep byte order");
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some("MySQL orders these by declaration index, not by label");
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL \
+ or not an ordered scalar (tinyint(1), JSON)"
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "row '{value}' is below the stored offset '{previous}' for tracking_column \
+ '{column}'. Filter the query with `{column} > $offset`"
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. Order the query ascending by '{column}'"
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+ /// Mark/delete work for a batch the runtime has already sent. It is persisted with
+ /// the batch's offsets so a crash between the send and the cleanup reruns it.
+ #[serde(default)]
+ pending_operations: Vec<PendingOperation>,
+}
+
+/// The rows of one table to mark or delete once their batch is acknowledged.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct PendingOperation {
+ table: String,
+ ids: Vec<PkValue>,
+ target: CleanupTarget,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct CleanupTarget {
+ key_column: String,
+ action: CleanupAction,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum CleanupAction {
+ Delete,
+ MarkProcessed { column: String },
+}
+
+/// The candidate state of the batch most recently handed to the runtime. Live state
+/// only takes it over on Ack, so a Nack leaves offsets and rows where they were.
+#[derive(Debug, Clone)]
+struct PendingBatch {
+ state: State,
+ /// The runtime has delivered the batch but its cleanup has not finished yet.
+ acknowledged: bool,
+ /// Cleanup finished and the next poll must persist `state` without the operations,
+ /// so a restart does not run them again.
+ retiring: bool,
+}
+
+struct PolledBatch {
+ messages: Vec<ProducedMessage>,
+ pending: Option<PendingBatch>,
+}
+
+enum CleanupKeyCheck {
+ Unique,
+ NotUnique,
+ TableMissing,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty, list the table even when custom_query hardcodes it"
+ .to_string(),
+ ));
+ }
+
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ if let Some(batch_size) = self.config.batch_size
+ && batch_size > MAX_CLEANUP_BATCH_SIZE
+ && self.cleanup_target().is_some()
+ {
+ return Err(Error::InitError(format!(
+ "batch_size {batch_size} exceeds {MAX_CLEANUP_BATCH_SIZE} with delete_after_read or \
+ processed_column set, MySQL allows at most {MAX_CLEANUP_BATCH_SIZE} placeholders \
+ in the statement that marks or deletes a batch"
+ )));
+ }
+
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "name the column custom_query orders by and filters with $offset"
+ } else {
+ "use a unique, monotonically increasing column"
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set, {guidance}"
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.validate_pending_cleanup()?;
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+ self.validate_cleanup_keys().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ if let Some(state) = self.retiring_state().await {
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: Some(self.serialize_pending_state(&state)?),
+ });
+ }
+
+ // Skip the pacing delay while draining a backlog, but never while a cleanup
+ // retry is pending, or a failing mark/delete would spin without a pause.
+ let cleanup_pending = self.pending_batch.lock().await.is_some();
+ if cleanup_pending || !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ {
+ let pending = self.pending_batch.lock().await;
+ if let Some(pending) = pending.as_ref() {
+ if !pending.acknowledged {
+ error!(
+ "{CONNECTOR_NAME} connector ID: {} was polled while a batch was still in flight",
+ self.id
+ );
+ return Err(Error::InvalidState);
+ }
+ // An empty batch whose Ack retries the cleanup the last one left behind.
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: None,
+ });
+ }
+ }
+
+ let polled = self.poll_tables().await?;
+
+ let processed_rows = match &polled.pending {
+ Some(pending) => pending.state.processed_rows,
+ None => self.state.lock().await.processed_rows,
+ };
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ }
+
+ // Idle cycles stage nothing, so return None to let the runtime skip the fsync.
+ let persisted_state = polled
+ .pending
+ .as_ref()
+ .map(|pending| self.serialize_pending_state(&pending.state))
+ .transpose()?;
+ *self.pending_batch.lock().await = polled.pending;
+
+ Ok(ProducedMessages {
+ schema,
+ messages: polled.messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> {
+ if result == SourceBatchResult::Nack {
+ // Dropping the candidate leaves the live offsets where they were, so the
+ // next poll reads the same rows again. Delivered batches keep their cleanup.
+ let mut pending = self.pending_batch.lock().await;
+ if pending
+ .as_ref()
+ .is_some_and(|pending| !pending.acknowledged && !pending.retiring)
+ {
+ pending.take();
+ }
+ return Ok(());
+ }
+
+ let Some(PendingBatch { mut state, .. }) = self.pending_batch.lock().await.clone() else {
+ return Ok(());
+ };
+
+ let operations = std::mem::take(&mut state.pending_operations);
+ let had_operations = !operations.is_empty();
+ let deadline = tokio::time::Instant::now() + ACK_BATCH_TIMEOUT;
+ let failed_operations = self.apply_pending_operations(operations, deadline).await?;
+
+ if failed_operations.is_empty() {
+ *self.state.lock().await = state.clone();
+ let mut pending = self.pending_batch.lock().await;
+ if had_operations {
+ *pending = Some(PendingBatch {
+ state,
+ acknowledged: false,
+ retiring: true,
+ });
+ } else {
+ pending.take();
+ }
+ } else {
+ state.pending_operations = failed_operations;
+ *self.pending_batch.lock().await = Some(PendingBatch {
+ state,
+ acknowledged: true,
+ retiring: false,
+ });
+ }
+
+ Ok(())
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ let state = restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ pending_operations: Vec::new(),
+ });
+ // The runtime only persists a batch's operations after sending it, so restored
+ // operations belong to a delivered batch and only the cleanup is left to do.
+ let pending_batch = (!state.pending_operations.is_empty()).then(|| PendingBatch {
+ state: state.clone(),
+ acknowledged: true,
+ retiring: false,
+ });
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(state),
+ pending_batch: Mutex::new(pending_batch),
+ consecutive_cleanup_failures: AtomicU32::new(0),
+ verified_cleanup_keys: Mutex::new(HashSet::new()),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ poisoned_tables: Mutex::new(HashSet::new()),
+ tracking_kinds: Mutex::new(HashMap::new()),
+ polls: AtomicU64::new(0),
+ cursor_regressions: Mutex::new(HashMap::new()),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ // `format_offset_value` escapes backslashes, which under NO_BACKSLASH_ESCAPES
+ // would reach MySQL as literal characters and filter on the wrong cursor.
+ .after_connect(|connection, _| {
+ Box::pin(async move {
+ sqlx::query(
+ "SET SESSION sql_mode = REPLACE(@@SESSION.sql_mode, 'NO_BACKSLASH_ESCAPES', '')",
+ )
+ .execute(&mut *connection)
+ .await?;
+ Ok(())
+ })
+ })
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn tracking_column(&self) -> &str {
+ self.config.tracking_column.as_deref().unwrap_or("id")
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn resolve_tracking_columns(&self) -> Result<HashMap<String, OffsetKind>, Error> {
+ // Whether the connector owns the query, and can therefore hold the schema to
+ // what that query will do with it.
+ let strict = self.config.custom_query.is_none();
+ let pool = self.get_pool()?;
+ let tracking_column = self.tracking_column();
+ let mut kinds = HashMap::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ }
+
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ }
+
+ warn!(
+ "tracking_column '{tracking_column}' is not a column of table '{table}'; \
+ assuming custom_query projects it. Its comparison order is resolved from \
+ the first result set instead."
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ // The column's type survives a plain projection, so it is worth rejecting
+ // whatever the query looks like. A custom_query aliasing an expression onto
+ // a name the table also uses is the one false positive, hence the hint.
+ if let Some(reason) = unusable_tracking_type(&data_type) {
+ let aliasing_hint = if self.config.custom_query.is_some() {
+ " If custom_query aliases a different expression to this name, rename the \
+ alias so it does not collide with the column."
+ } else {
+ ""
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': {reason}. Use an integer, timestamp, or string \
+ column.{aliasing_hint}"
+ )));
+ }
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; a row \
+ whose value is NULL cannot produce a cursor and will fail that table's batch. \
+ Have custom_query exclude those rows, or declare the column NOT NULL."
+ );
+ }
+
+ if is_collation_ordered(&data_type) {
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': values must be unique and increase in its collation order \
+ (zero-pad numbers, prefer a _bin collation)"
+ );
+ }
+
+ kinds.insert(table.clone(), offset_kind_for_data_type(&data_type));
+ }
+ Ok(kinds)
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
Review Comment:
removed the wrapper
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3554457860
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
Review Comment:
opened a sibling issue #3635
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3603747074
##########
core/connectors/sources/mysql_source/README.md:
##########
@@ -0,0 +1,424 @@
+# MySQL Source Connector
+
+The MySQL source connector fetches data from MySQL databases and streams it to Iggy topics. It supports incremental table polling with flexible payload extraction.
+
+> **Note:** Only polling mode is available. Binlog CDC support is planned for a future release.
+
+## Features
+
+- **Table Polling**: Incrementally fetch data from MySQL tables using a tracking column
+- **Flexible Payload Extraction**: Extract BLOB, TEXT, or JSON columns directly as payload
+- **Custom Queries**: Use custom SQL queries with parameter substitution
+- **Delete After Read**: Automatically delete rows after processing
+- **Mark as Processed**: Mark rows as processed using a boolean column
+- **Multiple Tables**: Monitor multiple tables simultaneously
+- **Batch Processing**: Fetch data in configurable batch sizes
+- **Offset Tracking**: Resume incremental polling from the last processed tracking value (see [Delivery semantics](#delivery-semantics))
+
+## Configuration
+
+```toml
+[plugin_config]
+connection_string = "mysql://user:pass@localhost:3306/database"
+tables = ["users", "orders"]
+poll_interval = "1s"
+batch_size = 1000
+tracking_column = "id"
+initial_offset = "0"
+max_connections = 10
+snake_case_columns = false
+include_metadata = true
+
+# Payload extraction (optional)
+payload_column = "payload"
+payload_format = "bytea"
+
+# Delete/mark processed (optional)
+delete_after_read = false
+processed_column = "is_processed"
+primary_key_column = "id"
+
+# Custom query (optional)
+custom_query = "SELECT * FROM $table WHERE id > $offset ORDER BY id LIMIT $limit"
+```
+
+## Configuration Options
+
+| Option | Type | Default | Description |
+| ------ | ---- | ------- | ----------- |
+| `connection_string` | string | required | MySQL connection string (`mysql://user:pass@host:3306/db`) |
+| `tables` | array | required | List of tables to monitor |
+| `poll_interval` | string | `10s` | How often to poll (e.g., `1s`, `5m`) |
+| `batch_size` | u32 | `1000` | Max rows per poll |
+| `tracking_column` | string | `id` | Column for incremental polling; must be unique and monotonically increasing (see [Tracking Column Requirements](#tracking-column-requirements)) |
+| `initial_offset` | string | none | Starting value for tracking column |
+| `max_connections` | u32 | `10` | Max database connections |
+| `snake_case_columns` | bool | `false` | Convert column names to snake_case |
+| `include_metadata` | bool | `true` | Wrap results with metadata envelope |
+| `payload_column` | string | none | Column to extract directly as payload |
+| `payload_format` | string | `json` (invalid if `payload_column` is set — `payload_format` is required in that case) | Format of payload_column: `bytea`, `text`, or `json_direct` |
+| `delete_after_read` | bool | `false` | Delete rows after reading |
+| `processed_column` | string | none | Boolean column to mark as processed |
+| `primary_key_column` | string | tracking_column | PK for delete/mark operations |
+| `custom_query` | string | none | Custom SQL with parameter substitution |
+| `verbose_logging` | bool | `false` | Log at info level instead of debug |
+| `max_retries` | u32 | `3` | Retries after the initial attempt for transient errors (3 = 4 total attempts) |
+| `retry_delay` | string | `1s` | Base delay between retries (e.g., `500ms`, `2s`) |
+
+## Tracking Column Requirements
+
+Polling is **insert-only**. Each poll runs roughly `SELECT ... WHERE tracking_column > last_offset ORDER BY tracking_column ASC LIMIT batch_size` and stores the largest tracking value it saw as the next offset. For this to be lossless the **`tracking_column` must be unique and monotonically increasing** — an auto-increment primary key is the canonical choice.
+
+The requirement is not cosmetic: if more than `batch_size` rows share the same tracking value, only the first `batch_size` are read and the next poll's `>` filter skips the rest of that value permanently. The same skip applies to `processed_column` and `delete_after_read`, because the tracking filter runs before those.
+
+For this reason a mutable, low-resolution column such as `updated_at` is **not** a safe tracking column: many rows commonly share the same second, and rows updated in place are never re-read. Capturing updates to existing rows is out of scope for polling.
+
+## Output Modes
+
+### JSON Mode (Default)
+
+When `payload_column` is not set, each row is wrapped in a `DatabaseRecord` JSON structure:
+
+```json
+{
+ "table_name": "users",
+ "operation_type": "SELECT",
+ "timestamp": "2024-01-15T10:30:00Z",
+ "data": {
+ "id": 123,
+ "name": "John Doe",
+ "email": "john@example.com"
+ },
+ "old_data": null
+}
+```
+
+The stream config should use `schema = "json"`.
+
+When `include_metadata = false`, the envelope is omitted and the row columns are serialized directly:
+
+```json
+{
+ "id": 123,
+ "name": "John Doe",
+ "email": "john@example.com"
+}
+```
+
+### Payload Column Extraction
+
+When `payload_column` is set, the connector extracts that column directly as the Iggy message payload. The `payload_format` option determines how the column is read:
+
+| Format | Column Type | Schema | Description |
+| ------ | ----------- | ------ | ----------- |
+| `bytea` / `raw` | `BLOB`, `BINARY`, `VARBINARY` | `raw` | Raw bytes passthrough |
+| `text` | `TEXT`, `VARCHAR` | `text` | UTF-8 text |
+| `json_direct` / `jsonb` | `JSON` | `json` | JSON object serialized to bytes |
Review Comment:
lib.rs:110-116 — README lists jsonbalias; code only acceptsjson_direct. **Fix**: add jsonb to from_config or drop from README.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3603861857
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1563 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ // Skip the pacing delay while draining a backlog
+ if !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ // Idle cycles produce no messages and leave offsets/processed_rows
+ // untouched, so return None to let the runtime skip the state fsync.
+ let persisted_state = if messages.is_empty() {
+ None
+ } else {
+ self.serialize_state(&state)
+ };
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+ let offset = format_offset_value(&offset_value);
+ let now = Utc::now();
+
Review Comment:
lib.rs:399 vs lib.rs:424 - custom_query $table unquoted; built-in path quotes. **Fix**: quote_qualified_identifier on $table (postgres same pattern).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3624941977
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1563 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ // Skip the pacing delay while draining a backlog
+ if !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ // Idle cycles produce no messages and leave offsets/processed_rows
+ // untouched, so return None to let the runtime skip the state fsync.
+ let persisted_state = if messages.is_empty() {
+ None
+ } else {
+ self.serialize_state(&state)
+ };
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+ let offset = format_offset_value(&offset_value);
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now_unix", &now.timestamp().to_string())
+ .replace("$now", &now.to_rfc3339())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
Review Comment:
implemented the typed PK encoder, added a `PKValue` enum, introduced a new function to extract the pk so that base64 encoding doesn't affect the mark/delete func. Also added bind logic inplace of splicing string literals in mark/delete func.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3568:
URL: https://github.com/apache/iggy/pull/3568#issuecomment-5335889602
This pull request was automatically closed because it has been inactive for 14 days. Feel free to reopen it if you'd like to continue working on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on PR #3568:
URL: https://github.com/apache/iggy/pull/3568#issuecomment-5993164229
/author
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r4186456308
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,3317 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some(
+ "at most 256 distinct values cannot be unique per row, and at display width 1 MySQL \
+ spells the type BOOLEAN, which reaches the connector as true/false with no ordered \
+ scalar form",
+ );
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some(
+ "binary values are carried as base64 text, which does not order the way the raw \
+ bytes do, and that text is what gets compared against the column",
+ );
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some(
+ "MySQL orders these by declaration index while only their labels reach the \
+ connector, and a bounded set of labels cannot be unique per row either",
+ );
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]. Every row's cursor \
+ comes from that column, so the same rows would be re-published on every poll while the \
+ stored offset never moved.",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL, or \
+ of a type with no ordered scalar form (tinyint(1), JSON). The cursor is taken from that \
+ value, so publishing the row would advance nothing and the same rows would be re-read on \
+ every poll."
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "a row arrived below the offset the poll resumed from by tracking_column \
+ '{column}': stored offset '{previous}', first row '{value}'. Taking the \
+ cursor from the last row would move it backwards. Filter the query on \
+ '{column}' with $offset so a batch cannot reach behind its own cursor."
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. The cursor is taken from the last row, so it would \
+ move backwards and rows would be skipped permanently. Order the query \
+ ascending by '{column}'."
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ // Every poll iterates `tables`, so an empty list makes the connector a
+ // silent no-op rather than an obvious misconfiguration.
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty. Even when custom_query hardcodes the table, list it \
+ here: the table name supplies the offset key, the deterministic message ID, \
+ and the table_name metadata field"
+ .to_string(),
+ ));
+ }
+
+ // A zero batch would make every fetch look like a full batch, so `poll`
+ // would drop its pacing sleep and spin on `LIMIT 0` queries forever.
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ // Only when the operator owns the ORDER BY. The built query sorts ascending by
+ // construction, so a one-row batch is safe there and legitimate for a slow table.
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset: a \
+ one-row batch offers no pair to compare, so the ordering check reduces to \
+ the row against the offset it resumed from, and a query ordered descending \
+ opens above that offset on every poll. The cursor would walk to the table's \
+ maximum and skip every row below it with nothing reporting it"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "the connector does not parse custom_query, so it has to be told which projected \
+ column supplies the cursor: name the one the query orders by and filters with \
+ $offset"
+ } else {
+ "it has to be unique and monotonically increasing under the order MySQL sorts it \
+ in."
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set: every row's cursor is taken from it, and {guidance}."
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ // Skip the pacing delay while draining a backlog
+ if !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ // Idle cycles produce no messages and leave offsets/processed_rows
+ // untouched, so return None to let the runtime skip the state fsync.
+ let persisted_state = if messages.is_empty() {
+ None
+ } else {
+ self.serialize_state(&state)
+ };
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ poisoned_tables: Mutex::new(HashSet::new()),
+ tracking_kinds: Mutex::new(HashMap::new()),
+ polls: AtomicU64::new(0),
+ cursor_regressions: Mutex::new(HashMap::new()),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ /// The column every cursor is read from. `open()` refuses a config that leaves it
+ /// unset, so the fallback is unreachable in a running connector.
+ fn tracking_column(&self) -> &str {
+ self.config.tracking_column.as_deref().unwrap_or("id")
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn resolve_tracking_columns(&self) -> Result<HashMap<String, OffsetKind>, Error> {
+ // Whether the connector owns the query, and can therefore hold the schema to
+ // what that query will do with it.
+ let strict = self.config.custom_query.is_none();
+ let pool = self.get_pool()?;
+ let tracking_column = self.tracking_column();
+ let mut kinds = HashMap::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ }
+
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ }
+
+ warn!(
+ "tracking_column '{tracking_column}' is not a column of table '{table}'; \
+ assuming custom_query projects it. Its comparison order is resolved from \
+ the first result set instead."
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
Review Comment:
Fixed in the latest push. poll_tables no longer writes to live state. It clones it into a candidate, advances tracking_offsets only on that clone, and stores it as the pending batch. Live state is committed in one place only on_batch_result on Ack. On Nack the candidate is dropped, so the next poll re-reads from the old offset. This applies with or without delete/mark configured.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r4306937649
##########
core/integration/tests/connectors/fixtures/mysql/source.rs:
##########
@@ -0,0 +1,1774 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::container::{
+ DEFAULT_TEST_STREAM, DEFAULT_TEST_TOPIC, ENV_SOURCE_CONNECTION_STRING, ENV_SOURCE_CUSTOM_QUERY,
+ ENV_SOURCE_DELETE_AFTER_READ, ENV_SOURCE_INCLUDE_METADATA, ENV_SOURCE_INITIAL_OFFSET,
+ ENV_SOURCE_PATH, ENV_SOURCE_PAYLOAD_COLUMN, ENV_SOURCE_PAYLOAD_FORMAT, ENV_SOURCE_PLUGIN_PATH,
+ ENV_SOURCE_POLL_INTERVAL, ENV_SOURCE_PRIMARY_KEY_COLUMN, ENV_SOURCE_PROCESSED_COLUMN,
+ ENV_SOURCE_STREAMS_0_SCHEMA, ENV_SOURCE_STREAMS_0_STREAM, ENV_SOURCE_STREAMS_0_TOPIC,
+ ENV_SOURCE_TABLES, ENV_SOURCE_TRACKING_COLUMN, MySqlContainer, MySqlOps, MySqlSourceOps,
+};
+use async_trait::async_trait;
+use integration::harness::{TestBinaryError, TestFixture};
+use sqlx::{MySql, Pool};
+use std::collections::HashMap;
+
+/// MySQL source fixture for JSON rows with metadata.
+///
+/// Creates a table with typed columns that get serialized as JSON with metadata.
+/// The boolean column is declared `BOOLEAN` (i.e. `tinyint(1)`) so `sqlx-mysql`
+/// reports it as `"BOOLEAN"` and the source emits JSON `true`/`false`, letting the
+/// shared `TestMessage { active: bool }` deserialize unchanged.
+pub struct MySqlSourceJsonFixture {
+ container: MySqlContainer,
+}
+
+impl MySqlOps for MySqlSourceJsonFixture {
+ fn container(&self) -> &MySqlContainer {
+ &self.container
+ }
+}
+
+impl MySqlSourceOps for MySqlSourceJsonFixture {
+ fn table_name(&self) -> &str {
+ Self::TABLE
+ }
+}
+
+impl MySqlSourceJsonFixture {
+ const TABLE: &'static str = "test_messages";
+
+ pub async fn create_table(&self, pool: &Pool<MySql>) {
+ let query = format!(
+ "CREATE TABLE IF NOT EXISTS `{}` (
+ id INT AUTO_INCREMENT PRIMARY KEY,
+ name VARCHAR(255) NOT NULL,
+ `count` INT NOT NULL,
+ amount DOUBLE NOT NULL,
+ active BOOLEAN NOT NULL,
+ `timestamp` BIGINT NOT NULL
+ )",
+ Self::TABLE
+ );
+ sqlx::query(sqlx::AssertSqlSafe(query))
+ .execute(pool)
+ .await
+ .unwrap_or_else(|e| panic!("Failed to create table: {e}"));
+ }
+
+ #[allow(clippy::too_many_arguments)]
+ pub async fn insert_row(
+ &self,
+ pool: &Pool<MySql>,
+ id: i32,
+ name: &str,
+ count: i32,
+ amount: f64,
+ active: bool,
+ timestamp: i64,
+ ) {
+ let query = format!(
+ "INSERT INTO `{}` (id, name, `count`, amount, active, `timestamp`) VALUES (?, ?, ?, ?, ?, ?)",
+ Self::TABLE
+ );
+ sqlx::query(sqlx::AssertSqlSafe(query))
+ .bind(id)
+ .bind(name)
+ .bind(count)
+ .bind(amount)
+ .bind(active)
+ .bind(timestamp)
+ .execute(pool)
+ .await
+ .unwrap_or_else(|e| panic!("Failed to insert row: {e}"));
+ }
+}
+
+#[async_trait]
+impl TestFixture for MySqlSourceJsonFixture {
+ async fn setup() -> Result<Self, TestBinaryError> {
+ let container = MySqlContainer::start().await?;
+ Ok(Self { container })
+ }
+
+ fn connectors_runtime_envs(&self) -> HashMap<String, String> {
Review Comment:
Every mysql_source fixture rebuilt the same connection, table, stream, topic, schema, poll interval and plugin path entries. They now start from one base_envs() helper and add only the settings that make each fixture distinct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on PR #3568:
URL: https://github.com/apache/iggy/pull/3568#issuecomment-5004298384
/author
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on PR #3568:
URL: https://github.com/apache/iggy/pull/3568#issuecomment-4826438910
/ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on PR #3568:
URL: https://github.com/apache/iggy/pull/3568#issuecomment-4928776638
Opened #3635 as the sibling issue for the missing runtime→source send ack, cross-linked with #2927/#2928/#2940. Left the current behavior in this PR as-is. @ryerraguntla @hubcio
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on PR #3568:
URL: https://github.com/apache/iggy/pull/3568#issuecomment-4928634345
@tusharagrahari - Overall looks good. Outside of the mysql source connector, two places needs to be updated for the newer connectors..
1. core/connectors/sources/README.md:9-14 — mysql_source missing from Available Sources table. Fix: add row.
2. .github/workflows/_build_rust_artifacts.yml:49 — iggy_connector_mysql_source absent from connector_plugins default list; edge releases ship no prebuilt .so. Postgres listed. Fix: register crate. (
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on PR #3568:
URL: https://github.com/apache/iggy/pull/3568#issuecomment-5879298599
closing due to inactivity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on PR #3568:
URL: https://github.com/apache/iggy/pull/3568#issuecomment-5980452368
@hubcio was stuck on something, will complete this in this iteration. Pls reopen the pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r4306818767
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,4071 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source,
+ source::SourceBatchResult, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+/// Upper bound on applying a batch's staged mark/delete after its Ack. The SDK waits
+/// `BATCH_RESULT_TIMEOUT` (30s) for the result, so this has to finish well inside it.
+const ACK_BATCH_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// MySQL rejects a prepared statement with more placeholders than this (error 1390), and
+/// the mark/delete binds one per row of the batch.
+const MAX_CLEANUP_BATCH_SIZE: u32 = 65_535;
+
+/// How many mark/delete statements in a row may fail before the connector stops. The
+/// operations stay in the persisted state and run again on restart.
+const MAX_CONSECUTIVE_CLEANUP_FAILURES: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ pending_batch: Mutex<Option<PendingBatch>>,
+ consecutive_cleanup_failures: AtomicU32,
+ verified_cleanup_keys: Mutex<HashSet<String>>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some("it has too few distinct values to be unique per row");
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some("binary values are compared as base64 text, which does not keep byte order");
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some("MySQL orders these by declaration index, not by label");
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL \
+ or not an ordered scalar (tinyint(1), JSON)"
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "row '{value}' is below the stored offset '{previous}' for tracking_column \
+ '{column}'. Filter the query with `{column} > $offset`"
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. Order the query ascending by '{column}'"
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+ /// Mark/delete work for a batch the runtime has already sent. It is persisted with
+ /// the batch's offsets so a crash between the send and the cleanup reruns it.
+ #[serde(default)]
+ pending_operations: Vec<PendingOperation>,
+}
+
+/// The rows of one table to mark or delete once their batch is acknowledged.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct PendingOperation {
+ table: String,
+ ids: Vec<PkValue>,
+ target: CleanupTarget,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct CleanupTarget {
+ key_column: String,
+ action: CleanupAction,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum CleanupAction {
+ Delete,
+ MarkProcessed { column: String },
+}
+
+/// The candidate state of the batch most recently handed to the runtime. Live state
+/// only takes it over on Ack, so a Nack leaves offsets and rows where they were.
+#[derive(Debug, Clone)]
+struct PendingBatch {
+ state: State,
+ /// The runtime has delivered the batch but its cleanup has not finished yet.
+ acknowledged: bool,
+ /// Cleanup finished and the next poll must persist `state` without the operations,
+ /// so a restart does not run them again.
+ retiring: bool,
+}
+
+struct PolledBatch {
+ messages: Vec<ProducedMessage>,
+ pending: Option<PendingBatch>,
+}
+
+enum CleanupKeyCheck {
+ Unique,
+ NotUnique,
+ TableMissing,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty, list the table even when custom_query hardcodes it"
+ .to_string(),
+ ));
+ }
+
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ if let Some(batch_size) = self.config.batch_size
+ && batch_size > MAX_CLEANUP_BATCH_SIZE
+ && self.cleanup_target().is_some()
+ {
+ return Err(Error::InitError(format!(
+ "batch_size {batch_size} exceeds {MAX_CLEANUP_BATCH_SIZE} with delete_after_read or \
+ processed_column set, MySQL allows at most {MAX_CLEANUP_BATCH_SIZE} placeholders \
+ in the statement that marks or deletes a batch"
+ )));
+ }
+
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "name the column custom_query orders by and filters with $offset"
+ } else {
+ "use a unique, monotonically increasing column"
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set, {guidance}"
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.validate_pending_cleanup()?;
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+ self.validate_cleanup_keys().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ if let Some(state) = self.retiring_state().await {
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: Some(self.serialize_pending_state(&state)?),
+ });
+ }
+
+ // Skip the pacing delay while draining a backlog, but never while a cleanup
+ // retry is pending, or a failing mark/delete would spin without a pause.
+ let cleanup_pending = self.pending_batch.lock().await.is_some();
+ if cleanup_pending || !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ {
+ let pending = self.pending_batch.lock().await;
+ if let Some(pending) = pending.as_ref() {
+ if !pending.acknowledged {
+ error!(
+ "{CONNECTOR_NAME} connector ID: {} was polled while a batch was still in flight",
+ self.id
+ );
+ return Err(Error::InvalidState);
+ }
+ // An empty batch whose Ack retries the cleanup the last one left behind.
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: None,
+ });
+ }
+ }
+
+ let polled = self.poll_tables().await?;
+
+ let processed_rows = match &polled.pending {
+ Some(pending) => pending.state.processed_rows,
+ None => self.state.lock().await.processed_rows,
+ };
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ }
+
+ // Idle cycles stage nothing, so return None to let the runtime skip the fsync.
+ let persisted_state = polled
+ .pending
+ .as_ref()
+ .map(|pending| self.serialize_pending_state(&pending.state))
+ .transpose()?;
+ *self.pending_batch.lock().await = polled.pending;
+
+ Ok(ProducedMessages {
+ schema,
+ messages: polled.messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> {
+ if result == SourceBatchResult::Nack {
+ // Dropping the candidate leaves the live offsets where they were, so the
+ // next poll reads the same rows again. Delivered batches keep their cleanup.
+ let mut pending = self.pending_batch.lock().await;
+ if pending
+ .as_ref()
+ .is_some_and(|pending| !pending.acknowledged && !pending.retiring)
+ {
+ pending.take();
+ }
+ return Ok(());
+ }
+
+ let Some(PendingBatch { mut state, .. }) = self.pending_batch.lock().await.clone() else {
+ return Ok(());
+ };
+
+ let operations = std::mem::take(&mut state.pending_operations);
+ let had_operations = !operations.is_empty();
+ let deadline = tokio::time::Instant::now() + ACK_BATCH_TIMEOUT;
+ let failed_operations = self.apply_pending_operations(operations, deadline).await?;
+
+ if failed_operations.is_empty() {
+ *self.state.lock().await = state.clone();
+ let mut pending = self.pending_batch.lock().await;
+ if had_operations {
+ *pending = Some(PendingBatch {
+ state,
+ acknowledged: false,
+ retiring: true,
+ });
+ } else {
+ pending.take();
+ }
+ } else {
+ state.pending_operations = failed_operations;
+ *self.pending_batch.lock().await = Some(PendingBatch {
+ state,
+ acknowledged: true,
+ retiring: false,
+ });
+ }
+
+ Ok(())
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ let state = restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ pending_operations: Vec::new(),
+ });
+ // The runtime only persists a batch's operations after sending it, so restored
+ // operations belong to a delivered batch and only the cleanup is left to do.
+ let pending_batch = (!state.pending_operations.is_empty()).then(|| PendingBatch {
+ state: state.clone(),
+ acknowledged: true,
+ retiring: false,
+ });
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(state),
+ pending_batch: Mutex::new(pending_batch),
+ consecutive_cleanup_failures: AtomicU32::new(0),
+ verified_cleanup_keys: Mutex::new(HashSet::new()),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ poisoned_tables: Mutex::new(HashSet::new()),
+ tracking_kinds: Mutex::new(HashMap::new()),
+ polls: AtomicU64::new(0),
+ cursor_regressions: Mutex::new(HashMap::new()),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ // `format_offset_value` escapes backslashes, which under NO_BACKSLASH_ESCAPES
+ // would reach MySQL as literal characters and filter on the wrong cursor.
+ .after_connect(|connection, _| {
+ Box::pin(async move {
+ sqlx::query(
+ "SET SESSION sql_mode = REPLACE(@@SESSION.sql_mode, 'NO_BACKSLASH_ESCAPES', '')",
+ )
+ .execute(&mut *connection)
+ .await?;
+ Ok(())
+ })
+ })
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn tracking_column(&self) -> &str {
+ self.config.tracking_column.as_deref().unwrap_or("id")
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn resolve_tracking_columns(&self) -> Result<HashMap<String, OffsetKind>, Error> {
+ // Whether the connector owns the query, and can therefore hold the schema to
+ // what that query will do with it.
+ let strict = self.config.custom_query.is_none();
+ let pool = self.get_pool()?;
+ let tracking_column = self.tracking_column();
+ let mut kinds = HashMap::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ }
+
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ }
+
+ warn!(
+ "tracking_column '{tracking_column}' is not a column of table '{table}'; \
+ assuming custom_query projects it. Its comparison order is resolved from \
+ the first result set instead."
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ // The column's type survives a plain projection, so it is worth rejecting
+ // whatever the query looks like. A custom_query aliasing an expression onto
+ // a name the table also uses is the one false positive, hence the hint.
+ if let Some(reason) = unusable_tracking_type(&data_type) {
+ let aliasing_hint = if self.config.custom_query.is_some() {
+ " If custom_query aliases a different expression to this name, rename the \
+ alias so it does not collide with the column."
+ } else {
+ ""
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': {reason}. Use an integer, timestamp, or string \
+ column.{aliasing_hint}"
+ )));
+ }
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; a row \
+ whose value is NULL cannot produce a cursor and will fail that table's batch. \
+ Have custom_query exclude those rows, or declare the column NOT NULL."
+ );
+ }
+
+ if is_collation_ordered(&data_type) {
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': values must be unique and increase in its collation order \
+ (zero-pad numbers, prefer a _bin collation)"
+ );
+ }
+
+ kinds.insert(table.clone(), offset_kind_for_data_type(&data_type));
+ }
+ Ok(kinds)
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn serialize_pending_state(&self, state: &State) -> Result<ConnectorState, Error> {
+ self.serialize_state(state).ok_or_else(|| {
+ Error::Serialization(format!("failed to serialize {CONNECTOR_NAME} state"))
+ })
+ }
+
+ async fn retiring_state(&self) -> Option<State> {
+ self.pending_batch
+ .lock()
+ .await
+ .as_ref()
+ .filter(|pending| pending.retiring)
+ .map(|pending| pending.state.clone())
+ }
+
+ fn primary_key_column(&self) -> &str {
+ self.config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(self.tracking_column())
+ }
+
+ fn cleanup_target(&self) -> Option<CleanupTarget> {
+ let action = if self.config.delete_after_read.unwrap_or(false) {
+ CleanupAction::Delete
+ } else if let Some(column) = &self.config.processed_column {
+ CleanupAction::MarkProcessed {
+ column: column.clone(),
+ }
+ } else {
+ return None;
+ };
+ Some(CleanupTarget {
+ key_column: self.primary_key_column().to_string(),
+ action,
+ })
+ }
+
+ /// Restored operations were staged under the config of the run that wrote them.
+ /// Running them under a different key or action would touch rows nobody read.
+ fn validate_pending_cleanup(&mut self) -> Result<(), Error> {
+ let configured = self.cleanup_target();
+ for operation in &self.state.get_mut().pending_operations {
+ if configured.as_ref() != Some(&operation.target) {
+ return Err(Error::InitError(format!(
+ "the cleanup configuration changed while cleanup of {} delivered row(s) in \
+ table '{}' is pending; restore primary_key_column '{}' and action {:?}",
+ operation.ids.len(),
+ operation.table,
+ operation.target.key_column,
+ operation.target.action
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ /// `WHERE key IN (...)` on a key that repeats marks or deletes every row sharing a
+ /// value, including ones that were never read. Tables that don't exist yet are
+ /// checked by the first poll that stages cleanup for them.
+ async fn validate_cleanup_keys(&mut self) -> Result<(), Error> {
+ let Some(target) = self.cleanup_target() else {
+ return Ok(());
+ };
+ let pool = self.get_pool()?;
+ let mut verified = HashSet::with_capacity(self.config.tables.len());
+ for table in &self.config.tables {
+ let check = cleanup_key_check(pool, table, &target.key_column)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to check primary_key_column '{}' on table '{table}': {e}",
+ target.key_column
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ verified.insert(table.clone());
+ }
+ CleanupKeyCheck::NotUnique => {
+ return Err(Error::InitError(not_unique_key_message(
+ table,
+ &target.key_column,
+ )));
+ }
+ CleanupKeyCheck::TableMissing => {}
+ }
+ }
+ *self.verified_cleanup_keys.get_mut() = verified;
+ Ok(())
+ }
+
+ async fn cleanup_key_is_unique(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ key_column: &str,
+ ) -> Result<bool, Error> {
+ if self.verified_cleanup_keys.lock().await.contains(table) {
+ return Ok(true);
+ }
+ let check = cleanup_key_check(pool, table, key_column)
+ .await
+ .map_err(|e| {
+ Error::Connection(format!(
+ "failed to check primary_key_column '{key_column}' on table '{table}': {e}"
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ self.verified_cleanup_keys
+ .lock()
+ .await
+ .insert(table.to_string());
+ Ok(true)
+ }
+ CleanupKeyCheck::NotUnique => Ok(false),
+ CleanupKeyCheck::TableMissing => Err(Error::Connection(format!(
+ "table '{table}' disappeared before its primary_key_column could be checked"
+ ))),
+ }
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ /// Best current knowledge of how a table's tracking values compare.
+ async fn tracking_kind(&self, table: &str) -> OffsetKind {
+ self.tracking_kinds
+ .lock()
+ .await
+ .get(table)
+ .copied()
+ .unwrap_or(OffsetKind::Unknown)
+ }
+
+ /// Whether the stored cursor is what limits the rows a poll can see. The built
+ /// polling query always filters on it; a `custom_query` does so only if it
+ /// interpolates `$offset`.
+ fn cursor_gates_query(&self) -> bool {
+ match &self.config.custom_query {
+ Some(query) => query.contains("$offset"),
+ None => true,
+ }
+ }
+
+ /// The offset a poll actually resumes from: the stored cursor, or the configured
+ /// starting point before anything has been stored.
+ fn effective_offset(&self, last_offset: &Option<String>) -> Option<String> {
+ last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let offset_value = self.effective_offset(last_offset).unwrap_or_default();
+ let offset = format_offset_value(&offset_value, kind);
+ let quoted_table = quote_qualified_identifier(table)?;
+ let limit = batch_size.to_string();
+ let now = Utc::now();
+ let now_unix = now.timestamp().to_string();
+ let now = now.to_rfc3339();
+ let replacements = [
+ ("$table", quoted_table.as_str()),
+ ("$offset", offset.as_str()),
+ ("$limit", limit.as_str()),
+ ("$now_unix", now_unix.as_str()),
+ ("$now", now.as_str()),
+ ];
+
+ Ok(substitute_query_tokens(query, &replacements))
+ }
+
+ /// The query is substituted as text and re-executed once per configured table on
+ /// every poll for the life of the process, so a shape that cannot terminate does
+ /// not fail, it repeats. What can be judged from the text alone is judged here,
+ /// at startup, rather than left to the first poll.
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let tracking_column = self.tracking_column();
+
+ match leading_statement_token(query) {
+ None => {
+ return Err(Error::InitError(
+ "custom_query has no statement: it is empty, or nothing but comments"
+ .to_string(),
+ ));
+ }
+ Some(token) if !READ_STATEMENT_STARTS.contains(&token.as_str()) => {
+ return Err(Error::InitError(format!(
+ "custom_query must be a read starting with SELECT, WITH or TABLE, found \
+ '{token}'"
+ )));
+ }
+ Some(_) => {}
+ }
+
+ // Collapsed so `ORDER\nBY` and `ORDER BY` read the same as `ORDER BY`; the
+ // check hard-fails, so a formatting difference must not decide it.
+ let normalized = query
+ .split_whitespace()
+ .collect::<Vec<_>>()
+ .join(" ")
+ .to_uppercase();
+
+ if query.contains("$offset") {
+ if !normalized.contains("ORDER BY") {
+ return Err(Error::InitError(format!(
+ "custom_query filters on $offset but has no ORDER BY, add \
+ `ORDER BY {tracking_column}`"
+ )));
+ }
+ } else if !self.config.delete_after_read.unwrap_or(false)
+ && self.config.processed_column.is_none()
+ {
+ return Err(Error::InitError(format!(
+ "custom_query must filter with $offset (`WHERE {tracking_column} > $offset`), or \
+ set delete_after_read or processed_column"
+ )));
+ }
+
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "custom_query uses $table but no tables are configured".to_string(),
+ ));
+ }
+ // Without $table the same SQL is executed once per table, so identical rows
+ // would be emitted repeatedly, each carrying a different table's metadata,
+ // message ID, and tracking offset. Table identity is not recoverable from a
+ // static query, so reject the combination instead of guessing one.
+ if !query.contains("$table") && self.config.tables.len() > 1 {
+ return Err(Error::InitError(format!(
+ "custom_query has no $table placeholder but {} tables are configured ({}), use \
+ $table in the query or configure a single table",
+ self.config.tables.len(),
+ self.config.tables.join(", ")
+ )));
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = self.effective_offset(last_offset) {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(&offset, kind)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn apply_pending_operations(
+ &self,
+ operations: Vec<PendingOperation>,
+ deadline: tokio::time::Instant,
+ ) -> Result<Vec<PendingOperation>, Error> {
+ let mut failed_operations = Vec::new();
+ let mut operations = operations.into_iter();
+ while let Some(operation) = operations.next() {
+ if tokio::time::Instant::now() >= deadline {
+ failed_operations.push(operation);
+ failed_operations.extend(operations);
+ break;
+ }
+ let result = tokio::time::timeout_at(deadline, async {
+ let pool = self.get_pool()?;
+ self.mark_or_delete_processed_rows(pool, &operation).await
+ })
+ .await;
+ match result {
+ Ok(Ok(())) => {
+ self.consecutive_cleanup_failures
+ .store(0, Ordering::Relaxed);
+ }
+ Ok(Err(error)) => {
+ self.record_cleanup_failure(&operation, &error)?;
+ failed_operations.push(operation);
+ }
+ Err(_) => {
+ self.record_cleanup_failure(
+ &operation,
+ "cleanup exceeded the 10s budget for one Ack",
+ )?;
+ failed_operations.push(operation);
+ }
+ }
+ }
+ Ok(failed_operations)
+ }
+
+ fn record_cleanup_failure(
+ &self,
+ operation: &PendingOperation,
+ error: impl std::fmt::Display,
+ ) -> Result<(), Error> {
+ let consecutive_failures = self
+ .consecutive_cleanup_failures
+ .fetch_add(1, Ordering::Relaxed)
+ .saturating_add(1);
+ error!(
+ "Failed to mark or delete {} delivered row(s) in table '{}' for {CONNECTOR_NAME} \
+ connector ID: {}, retrying on the next poll ({consecutive_failures} consecutive \
+ failures): {error}",
+ operation.ids.len(),
+ operation.table,
+ self.id
+ );
+ if consecutive_failures >= MAX_CONSECUTIVE_CLEANUP_FAILURES {
+ return Err(Error::Connection(format!(
+ "stopping {CONNECTOR_NAME} connector ID {} after {consecutive_failures} \
+ consecutive mark/delete failures; the cleanup stays in the saved state and \
+ runs again on restart",
+ self.id
+ )));
+ }
+ Ok(())
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ operation: &PendingOperation,
+ ) -> Result<(), Error> {
+ let PendingOperation { table, ids, target } = operation;
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(&target.key_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = match &target.action {
+ CleanupAction::Delete => {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
+ }
+ CleanupAction::MarkProcessed { column } => {
+ let quoted_processed = quote_identifier(column)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({placeholders})"
+ )
+ }
+ };
+
+ // The query and its binds are rebuilt on every attempt because with_retry
+ // calls the closure once per try and each `execute` consumes the query.
+ with_retry(
+ || {
+ let mut prepared = sqlx::query(sqlx::AssertSqlSafe(query.as_str()));
+ for id in ids {
+ prepared = match id {
+ PkValue::Text(text) => prepared.bind(text),
+ PkValue::Bytes(bytes) => prepared.bind(bytes),
+ };
+ }
+ prepared.execute(pool)
+ },
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<PolledBatch, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.tracking_column();
+ let pk_column = self.primary_key_column();
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ //
+ // An ordering violation is handled one step further. Retrying it would
+ // re-run the same query, re-decode the same rows, and hit the same failure
+ // forever, so the table is latched off instead of being retried each cycle.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ let poll_number = self.polls.fetch_add(1, Ordering::Relaxed) + 1;
+ let poisoned = { self.poisoned_tables.lock().await.clone() };
+ let mut newly_poisoned: Vec<String> = Vec::new();
+
+ for table in &self.config.tables {
+ if poisoned.contains(table) {
+ continue;
+ }
+
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ if !batch.messages.is_empty() {
+ self.cursor_regressions.lock().await.remove(table);
+ }
+ // A full batch means the LIMIT was hit, so more rows are waiting.
+ any_batch_full |= batch.messages.len() as u32 >= batch_size;
+ batches.push(batch);
+ }
+ Err(FetchError::Ordering(reason) | FetchError::MissingTrackingColumn(reason)) => {
+ error!(
+ "Table '{table}' at offset {}: {reason}. Table disabled until restart.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ newly_poisoned.push(table.clone());
+ }
+ Err(FetchError::CursorRegression(reason)) => {
+ // Cleared only by a batch that carried rows, so the run counts polls
+ // since the table last published anything. An idle poll produces as
+ // little as a rejected one does and must not read as recovery.
+ let consecutive = {
+ let mut regressions = self.cursor_regressions.lock().await;
+ let consecutive = regressions.entry(table.clone()).or_insert(0);
+ *consecutive += 1;
+ *consecutive
+ };
+ let offset = last_offset.as_deref().unwrap_or("<start>");
+ if consecutive >= CURSOR_REGRESSION_ERROR_THRESHOLD {
+ error!(
+ "Table '{table}' at offset {offset}: {reason}. Skipping this poll \
+ ({consecutive} consecutive polls)."
+ );
+ } else {
+ warn!("Table '{table}' at offset {offset}: {reason}. Skipping this poll.");
+ }
+ }
+ Err(FetchError::StaleOffsetKind(reason)) => {
+ warn!(
+ "Table '{table}' at offset {}: {reason}. Retrying on the next poll.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::UnusableTrackingValue(reason)) => {
+ error!(
+ "Table '{table}' at offset {}: {reason}. Retrying each poll until the row \
+ is fixed.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::Other(error)) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ if !newly_poisoned.is_empty() {
+ let mut poisoned_tables = self.poisoned_tables.lock().await;
+ poisoned_tables.extend(newly_poisoned);
+ if self
+ .config
+ .tables
+ .iter()
+ .all(|table| poisoned_tables.contains(table))
+ {
+ error!(
+ "All configured tables are disabled because their batches cannot yield a \
+ cursor; {CONNECTOR_NAME} connector with ID: {} is idle until restarted",
+ self.id
+ );
+ }
+ }
+
+ // From the next poll on a disabled table is skipped before anything is logged,
+ // so the one line written when it was disabled can scroll out of the operator's
+ // log window and leave a dark connector looking healthy.
+ if poll_number.is_multiple_of(DISABLED_TABLES_LOG_INTERVAL) {
+ let poisoned_tables = self.poisoned_tables.lock().await;
+ if !poisoned_tables.is_empty() {
+ let mut disabled: Vec<&str> = poisoned_tables.iter().map(String::as_str).collect();
+ disabled.sort_unstable();
+ error!(
+ "{} of {} configured table(s) are still disabled and are not being polled by \
+ {CONNECTOR_NAME} connector with ID: {}, and will stay that way until it is \
+ restarted: {}",
+ disabled.len(),
+ self.config.tables.len(),
+ self.id,
+ disabled.join(", ")
+ );
+ }
+ }
+
+ // Phase 2: stage each table's batch. Nothing touches MySQL or the live state
+ // here: the offsets and the mark/delete are applied in `on_batch_result` once
+ // the runtime has sent the batch and saved this candidate state, and a Nack
+ // throws the candidate away so the same rows are read again.
+ let cleanup_target = self.cleanup_target();
+ let mut candidate_state = self.state.lock().await.clone();
+ let mut operations = Vec::new();
+ let mut messages = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if let Some(target) = &cleanup_target
+ && !batch.messages.is_empty()
+ {
+ if batch.processed_ids.is_empty() {
+ error!(
+ "Table '{}': mark/delete is configured but no primary keys were extracted \
+ from {} row(s), skipping this cycle so rows are not published without \
+ being marked or deleted (check that '{pk_column}' is projected and scalar)",
+ batch.table,
+ batch.messages.len()
+ );
+ continue;
+ }
+
+ match self
+ .cleanup_key_is_unique(pool, &batch.table, &target.key_column)
+ .await
+ {
+ Ok(true) => {}
+ Ok(false) => {
+ error!(
+ "{}. Table disabled until restart.",
+ not_unique_key_message(&batch.table, &target.key_column)
+ );
+ self.poisoned_tables.lock().await.insert(batch.table);
+ continue;
+ }
+ Err(error) => {
+ error!("Table '{}' skipped this cycle: {error}", batch.table);
+ continue;
+ }
+ }
+
+ operations.push(PendingOperation {
+ table: batch.table.clone(),
+ ids: std::mem::take(&mut batch.processed_ids),
+ target: target.clone(),
+ });
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ candidate_state.tracking_offsets.insert(batch.table, offset);
+ }
+ }
+
+ let pending = (total_processed > 0).then(|| {
+ candidate_state.processed_rows += total_processed;
+ candidate_state.pending_operations = operations;
+ PendingBatch {
+ state: candidate_state,
+ acknowledged: false,
+ retiring: false,
+ }
+ });
+
+ self.last_batch_full
+ .store(any_batch_full, Ordering::Relaxed);
+ Ok(PolledBatch { messages, pending })
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, FetchError> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query_kind = self.tracking_kind(table).await;
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.substitute_query_params(custom_query, table, last_offset, batch_size, query_kind)?
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size, query_kind)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ // Settled once for the whole batch rather than per row, since the column list
+ // is a property of the result set.
+ if let Some(first) = rows.first() {
+ ensure_tracking_column_projected(
+ first.columns().iter().map(|column| column.name()),
+ tracking_column,
+ )?;
+ }
+
+ let resume_from = if self.cursor_gates_query() {
+ self.effective_offset(last_offset)
+ } else {
+ None
+ };
+
+ // The result set describes what this query actually returned, so it outranks
+ // whatever `information_schema` said at startup and is the only answer
+ // available for a computed column.
+ let kind = match rows.first().and_then(|row| {
Review Comment:
fixed. Now tables created after `open()` too goes through the same checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on PR #3568:
URL: https://github.com/apache/iggy/pull/3568#issuecomment-6253638841
Hi @hubcio, thanks for the review, i have made the necessary changes!
Pls re-review it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3568: feat(connectors): add MySQL source polling connector
URL: https://github.com/apache/iggy/pull/3568
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3568:
URL: https://github.com/apache/iggy/pull/3568#issuecomment-5137841018
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.
If you need a review, please ensure CI is green and the PR is rebased on the latest master. Don't hesitate to ping the maintainers - either `@core` on Discord or by mentioning them directly here on the PR.
Thank you for your contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3706999756
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.connect().await?;
+ self.validate_tracking_column().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ // Skip the pacing delay while draining a backlog
+ if !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ // Idle cycles produce no messages and leave offsets/processed_rows
+ // untouched, so return None to let the runtime skip the state fsync.
+ let persisted_state = if messages.is_empty() {
+ None
+ } else {
+ self.serialize_state(&state)
+ };
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn validate_tracking_column(&self) -> Result<(), Error> {
+ if self.config.custom_query.is_some() {
+ return Ok(());
+ }
+ let pool = self.get_pool()?;
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ continue;
+ }
+
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ if data_type.eq_ignore_ascii_case("json")
+ || column_type.eq_ignore_ascii_case("tinyint(1)")
+ {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type '{column_type}', \
+ which yields no ordered scalar cursor; use an integer, timestamp, or string column."
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+ let offset = format_offset_value(&offset_value);
+ let now = Utc::now();
+
+ let q = query
+ .replace("$table", "e_qualified_identifier(table)?)
+ .replace("$offset", &offset)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now_unix", &now.timestamp().to_string())
+ .replace("$now", &now.to_rfc3339());
+
+ Ok(q)
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[PkValue],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({placeholders})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ // The query and its binds are rebuilt on every attempt because with_retry
+ // calls the closure once per try and each `execute` consumes the query.
+ with_retry(
+ || {
+ let mut prepared = sqlx::query(sqlx::AssertSqlSafe(query.as_str()));
+ for id in ids {
+ prepared = match id {
+ PkValue::Text(text) => prepared.bind(text),
+ PkValue::Bytes(bytes) => prepared.bind(bytes),
+ };
+ }
+ prepared.execute(pool)
+ },
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ // A full batch means the LIMIT was hit, so more rows are waiting.
+ any_batch_full |= batch.messages.len() as u32 >= batch_size;
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. If the mark/delete call fails
+ // for a table we just skip its messages this cycle instead of queuing them
+ // - `mark_or_delete_processed_rows` already retries transient errors, so a
+ // failure here is permanent and the table's rows stay in MySQL to be
+ // retried next cycle, without blocking the others.
+ //
+ // Doesn't cover the case where mark/delete succeeds but the runtime never
+ // gets to publish the batch (crash, send failure, etc) - those rows are
+ // gone from MySQL with nothing delivered.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ let mark_or_delete = self.config.delete_after_read.unwrap_or(false)
+ || self.config.processed_column.is_some();
+
+ for mut batch in batches {
+ if mark_or_delete && !batch.messages.is_empty() && batch.processed_ids.is_empty() {
+ error!(
+ "Table '{}': mark/delete is configured but no primary keys were extracted \
+ from {} row(s), skipping this cycle so rows are not published without being \
+ marked or deleted (check that '{pk_column}' is projected and scalar)",
+ batch.table,
+ batch.messages.len()
+ );
+ continue;
+ }
+
+ if !batch.processed_ids.is_empty()
+ && let Err(error) = self
+ .mark_or_delete_processed_rows(
+ pool,
+ &batch.table,
+ pk_column,
+ &batch.processed_ids,
+ )
+ .await
+ {
+ error!(
+ "Failed to mark or delete processed rows for table '{}', skipping this cycle: {error}",
+ batch.table
+ );
+ continue;
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ state_updates.push((batch.table, offset));
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
+ }
+ }
+
+ self.last_batch_full
+ .store(any_batch_full, Ordering::Relaxed);
+ Ok(messages)
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, Error> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.substitute_query_params(custom_query, table, last_offset, batch_size)?
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ for row in rows {
+ let processed = self.process_row(&row, &table_config).map_err(|e| {
+ error!(
+ "Failed to decode row in table '{table}' (pk {}): {e}",
+ pk_for_log(&row, table_config.pk_column)
+ );
+ e
+ })?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+ if let Some(offset) = processed.max_offset {
+ batch.max_offset = Some(offset);
+ }
+
+ batch.messages.push(processed.message);
+ }
+
+ Ok(batch)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<PkValue> = None;
+ let mut max_offset: Option<String> = None;
+ let mut extracted_payload: Option<Vec<u8>> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
Review Comment:
`!config.payload_col.is_empty()` block now handles and publishes error (per table basis), when `extracted_payload.is_none()` even when `payload_col` is some.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3603736966
##########
core/connectors/sources/mysql_source/README.md:
##########
@@ -0,0 +1,424 @@
+# MySQL Source Connector
+
+The MySQL source connector fetches data from MySQL databases and streams it to Iggy topics. It supports incremental table polling with flexible payload extraction.
+
+> **Note:** Only polling mode is available. Binlog CDC support is planned for a future release.
+
+## Features
+
+- **Table Polling**: Incrementally fetch data from MySQL tables using a tracking column
+- **Flexible Payload Extraction**: Extract BLOB, TEXT, or JSON columns directly as payload
+- **Custom Queries**: Use custom SQL queries with parameter substitution
+- **Delete After Read**: Automatically delete rows after processing
+- **Mark as Processed**: Mark rows as processed using a boolean column
+- **Multiple Tables**: Monitor multiple tables simultaneously
+- **Batch Processing**: Fetch data in configurable batch sizes
+- **Offset Tracking**: Resume incremental polling from the last processed tracking value (see [Delivery semantics](#delivery-semantics))
+
+## Configuration
+
+```toml
+[plugin_config]
+connection_string = "mysql://user:pass@localhost:3306/database"
+tables = ["users", "orders"]
+poll_interval = "1s"
+batch_size = 1000
+tracking_column = "id"
+initial_offset = "0"
+max_connections = 10
+snake_case_columns = false
+include_metadata = true
+
+# Payload extraction (optional)
+payload_column = "payload"
+payload_format = "bytea"
+
+# Delete/mark processed (optional)
+delete_after_read = false
+processed_column = "is_processed"
+primary_key_column = "id"
+
+# Custom query (optional)
+custom_query = "SELECT * FROM $table WHERE id > $offset ORDER BY id LIMIT $limit"
+```
+
+## Configuration Options
+
+| Option | Type | Default | Description |
+| ------ | ---- | ------- | ----------- |
+| `connection_string` | string | required | MySQL connection string (`mysql://user:pass@host:3306/db`) |
+| `tables` | array | required | List of tables to monitor |
+| `poll_interval` | string | `10s` | How often to poll (e.g., `1s`, `5m`) |
+| `batch_size` | u32 | `1000` | Max rows per poll |
+| `tracking_column` | string | `id` | Column for incremental polling; must be unique and monotonically increasing (see [Tracking Column Requirements](#tracking-column-requirements)) |
+| `initial_offset` | string | none | Starting value for tracking column |
+| `max_connections` | u32 | `10` | Max database connections |
+| `snake_case_columns` | bool | `false` | Convert column names to snake_case |
+| `include_metadata` | bool | `true` | Wrap results with metadata envelope |
+| `payload_column` | string | none | Column to extract directly as payload |
+| `payload_format` | string | `json` (invalid if `payload_column` is set — `payload_format` is required in that case) | Format of payload_column: `bytea`, `text`, or `json_direct` |
+| `delete_after_read` | bool | `false` | Delete rows after reading |
+| `processed_column` | string | none | Boolean column to mark as processed |
+| `primary_key_column` | string | tracking_column | PK for delete/mark operations |
+| `custom_query` | string | none | Custom SQL with parameter substitution |
+| `verbose_logging` | bool | `false` | Log at info level instead of debug |
+| `max_retries` | u32 | `3` | Retries after the initial attempt for transient errors (3 = 4 total attempts) |
+| `retry_delay` | string | `1s` | Base delay between retries (e.g., `500ms`, `2s`) |
+
+## Tracking Column Requirements
+
+Polling is **insert-only**. Each poll runs roughly `SELECT ... WHERE tracking_column > last_offset ORDER BY tracking_column ASC LIMIT batch_size` and stores the largest tracking value it saw as the next offset. For this to be lossless the **`tracking_column` must be unique and monotonically increasing** — an auto-increment primary key is the canonical choice.
+
+The requirement is not cosmetic: if more than `batch_size` rows share the same tracking value, only the first `batch_size` are read and the next poll's `>` filter skips the rest of that value permanently. The same skip applies to `processed_column` and `delete_after_read`, because the tracking filter runs before those.
+
+For this reason a mutable, low-resolution column such as `updated_at` is **not** a safe tracking column: many rows commonly share the same second, and rows updated in place are never re-read. Capturing updates to existing rows is out of scope for polling.
+
+## Output Modes
+
+### JSON Mode (Default)
+
+When `payload_column` is not set, each row is wrapped in a `DatabaseRecord` JSON structure:
+
+```json
+{
+ "table_name": "users",
+ "operation_type": "SELECT",
+ "timestamp": "2024-01-15T10:30:00Z",
+ "data": {
+ "id": 123,
+ "name": "John Doe",
+ "email": "john@example.com"
+ },
+ "old_data": null
+}
+```
+
+The stream config should use `schema = "json"`.
+
+When `include_metadata = false`, the envelope is omitted and the row columns are serialized directly:
+
+```json
+{
+ "id": 123,
+ "name": "John Doe",
+ "email": "john@example.com"
+}
+```
+
+### Payload Column Extraction
+
+When `payload_column` is set, the connector extracts that column directly as the Iggy message payload. The `payload_format` option determines how the column is read:
+
+| Format | Column Type | Schema | Description |
+| ------ | ----------- | ------ | ----------- |
+| `bytea` / `raw` | `BLOB`, `BINARY`, `VARBINARY` | `raw` | Raw bytes passthrough |
+| `text` | `TEXT`, `VARCHAR` | `text` | UTF-8 text |
+| `json_direct` / `jsonb` | `JSON` | `json` | JSON object serialized to bytes |
+
+## Payload Format Examples
+
+### BLOB (Raw Bytes)
+
+Extract raw bytes from a BLOB column:
+
+```sql
+CREATE TABLE message_queue (
+ id INT AUTO_INCREMENT PRIMARY KEY,
+ payload BLOB NOT NULL
+);
+```
+
+```toml
+[[streams]]
+stream = "messages"
+topic = "queue"
+schema = "raw"
+batch_length = 100
+
+[plugin_config]
+tables = ["message_queue"]
+tracking_column = "id"
+payload_column = "payload"
+payload_format = "bytea"
+```
+
+### TEXT
+
+Extract text from a TEXT column:
+
+```sql
+CREATE TABLE logs (
+ id INT AUTO_INCREMENT PRIMARY KEY,
+ message TEXT NOT NULL
+);
+```
+
+```toml
+[[streams]]
+stream = "logs"
+topic = "app_logs"
+schema = "text"
+batch_length = 100
+
+[plugin_config]
+tables = ["logs"]
+tracking_column = "id"
+payload_column = "message"
+payload_format = "text"
+```
+
+### JSON (Direct)
+
+Extract a JSON column directly as JSON payload (without `DatabaseRecord` wrapper):
+
+```sql
+CREATE TABLE events (
+ id INT AUTO_INCREMENT PRIMARY KEY,
+ data JSON NOT NULL
+);
+```
+
+```toml
+[[streams]]
+stream = "events"
+topic = "user_events"
+schema = "json"
+batch_length = 100
+
+[plugin_config]
+tables = ["events"]
+tracking_column = "id"
+payload_column = "data"
+payload_format = "json_direct"
+```
+
+## Custom Query Parameters
+
+When using `custom_query`, these placeholders are available:
+
+| Placeholder | Replaced With |
+| ----------- | ------------- |
+| `$table` | Current table name |
+| `$offset` | Last processed offset (or `initial_offset`) |
+| `$limit` | `batch_size` value |
+| `$now` | Current UTC timestamp (RFC3339) |
+| `$now_unix` | Current Unix timestamp (seconds) |
+
+If the query uses `$offset`, it must be ordered by the tracking column in ascending order (`ORDER BY <tracking_column>` — MySQL sorts ascending by default, so `ASC` doesn't need to be spelled out). The connector takes the tracking value of the *last row returned* as the next `$offset`; without ascending order, that's not guaranteed to be the max, so rows can be skipped or re-emitted on the next poll.
+
+The column you order by is your effective tracking column, so the same [Tracking Column Requirements](#tracking-column-requirements) apply: it must be unique and monotonically increasing, or rows sharing a value can be skipped across a batch boundary.
+
+Example:
+
+```sql
+SELECT * FROM $table
+WHERE id > $offset
+ AND (scheduled_at IS NULL OR scheduled_at <= '$now')
+ORDER BY id
+LIMIT $limit
+```
+
+## Delete After Read / Mark as Processed
+
+### Delete After Read
+
+Deletes rows from the source table after successful processing. At-most-once — see [Delivery semantics](#delivery-semantics) below.
+
+```toml
+[plugin_config]
+delete_after_read = true
+primary_key_column = "id"
+```
+
+### Mark as Processed
+
+Updates a boolean column instead of deleting. At-most-once — see [Delivery semantics](#delivery-semantics) below.
+
+```toml
+[plugin_config]
+processed_column = "is_processed"
+primary_key_column = "id"
+```
+
+Your table needs the boolean column:
+
+```sql
+ALTER TABLE users ADD COLUMN is_processed BOOLEAN DEFAULT false;
+```
+
+When `processed_column` is set, the connector automatically adds a `WHERE is_processed = FALSE` filter to the polling query, so only unprocessed rows are fetched.
+
+### Delivery semantics
+
+Both `delete_after_read` and `processed_column` mutate MySQL during the poll, before the batch is actually sent to Iggy. If the connector crashes or the send fails in that window, those rows are gone or marked processed anyway — they won't be retried. So both options are at-most-once, not at-least-once.
Review Comment:
>> those rows are gone or marked processed anyway - will it be deleted (gone) or marked? It should be either one in each pass, could this be confirmed and update the documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3603820903
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1563 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ // Skip the pacing delay while draining a backlog
+ if !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ // Idle cycles produce no messages and leave offsets/processed_rows
+ // untouched, so return None to let the runtime skip the state fsync.
+ let persisted_state = if messages.is_empty() {
+ None
+ } else {
+ self.serialize_state(&state)
+ };
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+ let offset = format_offset_value(&offset_value);
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now_unix", &now.timestamp().to_string())
+ .replace("$now", &now.to_rfc3339())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() || id.parse::<u64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).execute(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ // A full batch means the LIMIT was hit, so more rows are waiting.
+ any_batch_full |= batch.messages.len() as u32 >= batch_size;
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. If the mark/delete call fails
+ // for a table we just skip its messages this cycle instead of queuing them
+ // - `mark_or_delete_processed_rows` already retries transient errors, so a
+ // failure here is permanent and the table's rows stay in MySQL to be
+ // retried next cycle, without blocking the others.
+ //
+ // Doesn't cover the case where mark/delete succeeds but the runtime never
+ // gets to publish the batch (crash, send failure, etc) - those rows are
+ // gone from MySQL with nothing delivered. That's just how
+ // delete_after_read/processed_column work, not something this loop fixes.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ let mark_or_delete = self.config.delete_after_read.unwrap_or(false)
+ || self.config.processed_column.is_some();
+
+ for mut batch in batches {
+ if mark_or_delete && !batch.messages.is_empty() && batch.processed_ids.is_empty() {
+ warn!(
+ "Table '{}': mark/delete is configured but no primary keys were extracted \
+ from {} row(s); rows publish without being marked or deleted \
+ (check that '{pk_column}' is projected and scalar)",
+ batch.table,
+ batch.messages.len()
+ );
+ }
+
+ if !batch.processed_ids.is_empty()
+ && let Err(error) = self
+ .mark_or_delete_processed_rows(
+ pool,
+ &batch.table,
+ pk_column,
+ &batch.processed_ids,
+ )
+ .await
+ {
+ error!(
+ "Failed to mark or delete processed rows for table '{}', skipping this cycle: {error}",
+ batch.table
+ );
+ continue;
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ state_updates.push((batch.table, offset));
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
+ }
+ }
+
+ self.last_batch_full
+ .store(any_batch_full, Ordering::Relaxed);
+ Ok(messages)
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, Error> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.substitute_query_params(custom_query, table, last_offset, batch_size)
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ for row in rows {
+ let processed = self.process_row(&row, &table_config).map_err(|e| {
+ error!(
+ "Failed to decode row in table '{table}' (pk {}): {e}",
+ pk_for_log(&row, table_config.pk_column)
+ );
+ e
+ })?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+ if let Some(offset) = processed.max_offset {
+ batch.max_offset = Some(offset);
+ }
+
+ batch.messages.push(processed.message);
+ }
+
+ Ok(batch)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<String> = None;
+ let mut max_offset: Option<String> = None;
+ let mut extracted_payload: Option<Vec<u8>> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ if name == config.payload_col {
+ extracted_payload =
+ Some(self.extract_payload_column(row, i, config.payload_format)?);
+ }
+ if name == config.tracking_column || name == config.pk_column {
+ let as_string = value_as_string(&extract_column_value(row, i)?);
Review Comment:
value_as_string only String|Number; BOOL/NULL/JSON tracking/PK ⇒ no offset/PK for mark. **Fix**: canonical scalar encoding or open-time schema probe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r4186485769
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,3317 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some(
+ "at most 256 distinct values cannot be unique per row, and at display width 1 MySQL \
+ spells the type BOOLEAN, which reaches the connector as true/false with no ordered \
+ scalar form",
+ );
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some(
+ "binary values are carried as base64 text, which does not order the way the raw \
+ bytes do, and that text is what gets compared against the column",
+ );
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some(
+ "MySQL orders these by declaration index while only their labels reach the \
+ connector, and a bounded set of labels cannot be unique per row either",
+ );
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]. Every row's cursor \
+ comes from that column, so the same rows would be re-published on every poll while the \
+ stored offset never moved.",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL, or \
+ of a type with no ordered scalar form (tinyint(1), JSON). The cursor is taken from that \
+ value, so publishing the row would advance nothing and the same rows would be re-read on \
+ every poll."
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "a row arrived below the offset the poll resumed from by tracking_column \
+ '{column}': stored offset '{previous}', first row '{value}'. Taking the \
+ cursor from the last row would move it backwards. Filter the query on \
+ '{column}' with $offset so a batch cannot reach behind its own cursor."
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. The cursor is taken from the last row, so it would \
+ move backwards and rows would be skipped permanently. Order the query \
+ ascending by '{column}'."
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ // Every poll iterates `tables`, so an empty list makes the connector a
+ // silent no-op rather than an obvious misconfiguration.
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty. Even when custom_query hardcodes the table, list it \
+ here: the table name supplies the offset key, the deterministic message ID, \
+ and the table_name metadata field"
+ .to_string(),
+ ));
+ }
+
+ // A zero batch would make every fetch look like a full batch, so `poll`
+ // would drop its pacing sleep and spin on `LIMIT 0` queries forever.
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ // Only when the operator owns the ORDER BY. The built query sorts ascending by
+ // construction, so a one-row batch is safe there and legitimate for a slow table.
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset: a \
+ one-row batch offers no pair to compare, so the ordering check reduces to \
+ the row against the offset it resumed from, and a query ordered descending \
+ opens above that offset on every poll. The cursor would walk to the table's \
+ maximum and skip every row below it with nothing reporting it"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "the connector does not parse custom_query, so it has to be told which projected \
+ column supplies the cursor: name the one the query orders by and filters with \
+ $offset"
+ } else {
+ "it has to be unique and monotonically increasing under the order MySQL sorts it \
+ in."
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set: every row's cursor is taken from it, and {guidance}."
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ // Skip the pacing delay while draining a backlog
+ if !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ // Idle cycles produce no messages and leave offsets/processed_rows
+ // untouched, so return None to let the runtime skip the state fsync.
+ let persisted_state = if messages.is_empty() {
Review Comment:
Fixed. Every state handed to the runtime now goes through serialize_pending_state, which maps a failed serialization to Err(Error::Serialization) like postgres_source, so the poll fails instead of returning state: None.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r4306845887
##########
core/connectors/sources/mysql_source/README.md:
##########
@@ -0,0 +1,631 @@
+# MySQL Source Connector
+
+The MySQL source connector fetches data from MySQL databases and streams it to Iggy topics. It supports incremental table polling with flexible payload extraction.
+
+> **Note:** Only polling mode is available. Binlog CDC support is planned for a future release.
+
+## Features
+
+- **Table Polling**: Incrementally fetch data from MySQL tables using a tracking column
+- **Flexible Payload Extraction**: Extract BLOB, TEXT, or JSON columns directly as payload
+- **Custom Queries**: Use custom SQL queries with parameter substitution
+- **Delete After Read**: Automatically delete rows after processing
+- **Mark as Processed**: Mark rows as processed using a boolean column
+- **Multiple Tables**: Monitor multiple tables, polled one after another each cycle
+- **Batch Processing**: Fetch data in configurable batch sizes
+- **Offset Tracking**: Resume incremental polling from the last processed tracking value (see [Delivery semantics](#delivery-semantics))
+
+## Configuration
+
+```toml
+[plugin_config]
+connection_string = "mysql://user:pass@localhost:3306/database"
+tables = ["users", "orders"]
+poll_interval = "1s"
+batch_size = 1000
+tracking_column = "id"
+initial_offset = "0"
+max_connections = 10
+snake_case_columns = false
+include_metadata = true
+
+# Payload extraction (optional)
+payload_column = "payload"
+payload_format = "bytea"
+
+# Delete/mark processed (optional)
+delete_after_read = false
+processed_column = "is_processed"
+primary_key_column = "id"
+
+# Custom query (optional)
+custom_query = "SELECT * FROM $table WHERE id > $offset ORDER BY id LIMIT $limit"
+```
+
+## Configuration Options
+
+| Option | Type | Default | Description |
+| ------ | ---- | ------- | ----------- |
+| `connection_string` | string | required | MySQL connection string (`mysql://user:pass@host:3306/db`) |
+| `tables` | array | required | List of tables to monitor |
+| `poll_interval` | string | `10s` | How often to poll (e.g., `1s`, `5m`) |
+| `batch_size` | u32 | `1000` | Max rows per poll; must be greater than 0, greater than 1 when `custom_query` filters on `$offset`, and at most 65535 when `delete_after_read` or `processed_column` is set (see [Ordering is checked on the rows](#ordering-is-checked-on-the-rows)) |
+| `tracking_column` | string | required | Column for incremental polling; must be unique and monotonically increasing (see [Tracking Column Requirements](#tracking-column-requirements)) |
+| `initial_offset` | string | none | Starting value for tracking column |
+| `max_connections` | u32 | `10` | Max database connections |
+| `snake_case_columns` | bool | `false` | Convert column names to snake_case |
+| `include_metadata` | bool | `true` | Wrap results with metadata envelope |
+| `payload_column` | string | none | Column to extract directly as payload |
+| `payload_format` | string | `json` (invalid if `payload_column` is set — `payload_format` is required in that case) | Format of payload_column: `bytea`, `text`, or `json_direct` |
+| `delete_after_read` | bool | `false` | Delete rows after reading |
+| `processed_column` | string | none | Boolean column to mark as processed |
+| `primary_key_column` | string | tracking_column | Key for delete/mark operations; must back a single-column `PRIMARY KEY` or `UNIQUE` index (see [Delete After Read / Mark as Processed](#delete-after-read--mark-as-processed)) |
+| `custom_query` | string | none | Custom SQL with parameter substitution; must be a read, and must either filter on `$offset` with an `ORDER BY` or consume its rows (see [What the query has to be](#what-the-query-has-to-be)) |
+| `verbose_logging` | bool | `false` | Log at info level instead of debug |
+| `max_retries` | u32 | `3` | Retries after the initial attempt for transient errors (3 = 4 total attempts) |
+| `retry_delay` | string | `1s` | Base delay between retries (e.g., `500ms`, `2s`) |
+
+## Tracking Column Requirements
+
+Polling is **insert-only**. Each poll runs roughly `SELECT ... WHERE tracking_column > last_offset ORDER BY tracking_column ASC LIMIT batch_size` and stores the largest tracking value it saw as the next offset. For this to be lossless the **`tracking_column` must be unique and monotonically increasing under the order MySQL sorts that column in** — an auto-increment primary key is the canonical choice.
Review Comment:
fixed in readme
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r4302519539
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,4071 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source,
+ source::SourceBatchResult, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+/// Upper bound on applying a batch's staged mark/delete after its Ack. The SDK waits
+/// `BATCH_RESULT_TIMEOUT` (30s) for the result, so this has to finish well inside it.
+const ACK_BATCH_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// MySQL rejects a prepared statement with more placeholders than this (error 1390), and
+/// the mark/delete binds one per row of the batch.
+const MAX_CLEANUP_BATCH_SIZE: u32 = 65_535;
+
+/// How many mark/delete statements in a row may fail before the connector stops. The
+/// operations stay in the persisted state and run again on restart.
+const MAX_CONSECUTIVE_CLEANUP_FAILURES: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ pending_batch: Mutex<Option<PendingBatch>>,
+ consecutive_cleanup_failures: AtomicU32,
+ verified_cleanup_keys: Mutex<HashSet<String>>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some("it has too few distinct values to be unique per row");
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some("binary values are compared as base64 text, which does not keep byte order");
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some("MySQL orders these by declaration index, not by label");
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL \
+ or not an ordered scalar (tinyint(1), JSON)"
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "row '{value}' is below the stored offset '{previous}' for tracking_column \
+ '{column}'. Filter the query with `{column} > $offset`"
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. Order the query ascending by '{column}'"
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+ /// Mark/delete work for a batch the runtime has already sent. It is persisted with
+ /// the batch's offsets so a crash between the send and the cleanup reruns it.
+ #[serde(default)]
+ pending_operations: Vec<PendingOperation>,
+}
+
+/// The rows of one table to mark or delete once their batch is acknowledged.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct PendingOperation {
+ table: String,
+ ids: Vec<PkValue>,
+ target: CleanupTarget,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct CleanupTarget {
+ key_column: String,
+ action: CleanupAction,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum CleanupAction {
+ Delete,
+ MarkProcessed { column: String },
+}
+
+/// The candidate state of the batch most recently handed to the runtime. Live state
+/// only takes it over on Ack, so a Nack leaves offsets and rows where they were.
+#[derive(Debug, Clone)]
+struct PendingBatch {
+ state: State,
+ /// The runtime has delivered the batch but its cleanup has not finished yet.
+ acknowledged: bool,
+ /// Cleanup finished and the next poll must persist `state` without the operations,
+ /// so a restart does not run them again.
+ retiring: bool,
+}
+
+struct PolledBatch {
+ messages: Vec<ProducedMessage>,
+ pending: Option<PendingBatch>,
+}
+
+enum CleanupKeyCheck {
+ Unique,
+ NotUnique,
+ TableMissing,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty, list the table even when custom_query hardcodes it"
+ .to_string(),
+ ));
+ }
+
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ if let Some(batch_size) = self.config.batch_size
+ && batch_size > MAX_CLEANUP_BATCH_SIZE
+ && self.cleanup_target().is_some()
+ {
+ return Err(Error::InitError(format!(
+ "batch_size {batch_size} exceeds {MAX_CLEANUP_BATCH_SIZE} with delete_after_read or \
+ processed_column set, MySQL allows at most {MAX_CLEANUP_BATCH_SIZE} placeholders \
+ in the statement that marks or deletes a batch"
+ )));
+ }
+
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "name the column custom_query orders by and filters with $offset"
+ } else {
+ "use a unique, monotonically increasing column"
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set, {guidance}"
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.validate_pending_cleanup()?;
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+ self.validate_cleanup_keys().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ if let Some(state) = self.retiring_state().await {
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: Some(self.serialize_pending_state(&state)?),
+ });
+ }
+
+ // Skip the pacing delay while draining a backlog, but never while a cleanup
+ // retry is pending, or a failing mark/delete would spin without a pause.
+ let cleanup_pending = self.pending_batch.lock().await.is_some();
+ if cleanup_pending || !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ {
+ let pending = self.pending_batch.lock().await;
+ if let Some(pending) = pending.as_ref() {
+ if !pending.acknowledged {
+ error!(
+ "{CONNECTOR_NAME} connector ID: {} was polled while a batch was still in flight",
+ self.id
+ );
+ return Err(Error::InvalidState);
+ }
+ // An empty batch whose Ack retries the cleanup the last one left behind.
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: None,
+ });
+ }
+ }
+
+ let polled = self.poll_tables().await?;
+
+ let processed_rows = match &polled.pending {
+ Some(pending) => pending.state.processed_rows,
+ None => self.state.lock().await.processed_rows,
+ };
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ }
+
+ // Idle cycles stage nothing, so return None to let the runtime skip the fsync.
+ let persisted_state = polled
+ .pending
+ .as_ref()
+ .map(|pending| self.serialize_pending_state(&pending.state))
+ .transpose()?;
+ *self.pending_batch.lock().await = polled.pending;
+
+ Ok(ProducedMessages {
+ schema,
+ messages: polled.messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> {
+ if result == SourceBatchResult::Nack {
+ // Dropping the candidate leaves the live offsets where they were, so the
+ // next poll reads the same rows again. Delivered batches keep their cleanup.
+ let mut pending = self.pending_batch.lock().await;
+ if pending
+ .as_ref()
+ .is_some_and(|pending| !pending.acknowledged && !pending.retiring)
+ {
+ pending.take();
+ }
+ return Ok(());
+ }
+
+ let Some(PendingBatch { mut state, .. }) = self.pending_batch.lock().await.clone() else {
+ return Ok(());
+ };
+
+ let operations = std::mem::take(&mut state.pending_operations);
+ let had_operations = !operations.is_empty();
+ let deadline = tokio::time::Instant::now() + ACK_BATCH_TIMEOUT;
+ let failed_operations = self.apply_pending_operations(operations, deadline).await?;
+
+ if failed_operations.is_empty() {
+ *self.state.lock().await = state.clone();
+ let mut pending = self.pending_batch.lock().await;
+ if had_operations {
+ *pending = Some(PendingBatch {
+ state,
+ acknowledged: false,
+ retiring: true,
+ });
+ } else {
+ pending.take();
+ }
+ } else {
+ state.pending_operations = failed_operations;
+ *self.pending_batch.lock().await = Some(PendingBatch {
+ state,
+ acknowledged: true,
+ retiring: false,
+ });
+ }
+
+ Ok(())
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ let state = restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ pending_operations: Vec::new(),
+ });
+ // The runtime only persists a batch's operations after sending it, so restored
+ // operations belong to a delivered batch and only the cleanup is left to do.
+ let pending_batch = (!state.pending_operations.is_empty()).then(|| PendingBatch {
+ state: state.clone(),
+ acknowledged: true,
+ retiring: false,
+ });
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(state),
+ pending_batch: Mutex::new(pending_batch),
+ consecutive_cleanup_failures: AtomicU32::new(0),
+ verified_cleanup_keys: Mutex::new(HashSet::new()),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ poisoned_tables: Mutex::new(HashSet::new()),
+ tracking_kinds: Mutex::new(HashMap::new()),
+ polls: AtomicU64::new(0),
+ cursor_regressions: Mutex::new(HashMap::new()),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ // `format_offset_value` escapes backslashes, which under NO_BACKSLASH_ESCAPES
+ // would reach MySQL as literal characters and filter on the wrong cursor.
+ .after_connect(|connection, _| {
+ Box::pin(async move {
+ sqlx::query(
+ "SET SESSION sql_mode = REPLACE(@@SESSION.sql_mode, 'NO_BACKSLASH_ESCAPES', '')",
+ )
+ .execute(&mut *connection)
+ .await?;
+ Ok(())
+ })
+ })
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn tracking_column(&self) -> &str {
+ self.config.tracking_column.as_deref().unwrap_or("id")
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn resolve_tracking_columns(&self) -> Result<HashMap<String, OffsetKind>, Error> {
+ // Whether the connector owns the query, and can therefore hold the schema to
+ // what that query will do with it.
+ let strict = self.config.custom_query.is_none();
+ let pool = self.get_pool()?;
+ let tracking_column = self.tracking_column();
+ let mut kinds = HashMap::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
Review Comment:
Fixed.
A table that doesn't exist at startup is now checked on the first poll after it appears, using the same information_schema probe open() runs. A rejected type (BINARY/BLOB, JSON, TINYINT, ENUM/SET) disables that table before anything is read, so the cursor never moves. The probe also resolves the column's comparison order before the first query, which fixes a related stall: a late text table was filtered with a bare code > 0, which matches no non-numeric value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3531355259
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
Review Comment:
will update the docs accordingly. For the tiebreak to work correctly we would have to make the pk monotonic anyway, and if an operator already has a unique monotonic pk column, there's no reason to track on a different column. This issue will be fullproof fixed with binlog CDC(once polling is done)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3554119014
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).execute(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. A table's messages are emitted
+ // only after its rows are marked or deleted, so a failure here can never
+ // leave a table marked-but-unpublished. `mark_or_delete_processed_rows`
+ // retries transient errors; a permanent failure isolates that one table
+ // (its rows stay in MySQL and are retried next cycle) while the others
+ // still flow.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if !batch.processed_ids.is_empty()
+ && let Err(error) = self
+ .mark_or_delete_processed_rows(
+ pool,
+ &batch.table,
+ pk_column,
+ &batch.processed_ids,
+ )
+ .await
+ {
+ error!(
+ "Failed to mark or delete processed rows for table '{}', skipping this cycle: {error}",
+ batch.table
+ );
+ continue;
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ state_updates.push((batch.table, offset));
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
+ }
+ state.last_poll_time = Utc::now();
+ }
+
+ Ok(messages)
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, Error> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.validate_custom_query(custom_query)?;
+ self.substitute_query_params(custom_query, table, last_offset, batch_size)
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ for row in rows {
+ let processed = self.process_row(&row, &table_config)?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+ if let Some(offset) = processed.max_offset {
+ batch.max_offset = Some(offset);
+ }
+
+ batch.messages.push(processed.message);
+ }
+
+ Ok(batch)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<String> = None;
+ let mut max_offset: Option<String> = None;
+ let mut extracted_payload: Option<Vec<u8>> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ if name == config.payload_col {
+ extracted_payload =
+ Some(self.extract_payload_column(row, i, config.payload_format)?);
+ }
+ if name == config.tracking_column {
+ max_offset = value_as_string(&extract_column_value(row, i)?);
+ }
+ if name == config.pk_column {
+ row_pk = value_as_string(&extract_column_value(row, i)?);
+ }
+ }
+ }
+
+ if extracted_payload.is_none() {
+ let mut data = serde_json::Map::new();
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ let column_name = if config.snake_case_columns {
+ to_snake_case(name)
+ } else {
+ name.to_string()
+ };
+ let value = extract_column_value(row, i)?;
+ if name == config.tracking_column {
+ max_offset = value_as_string(&value);
+ }
+ if name == config.pk_column {
+ row_pk = value_as_string(&value);
+ }
+ data.insert(column_name, value);
+ }
+
+ extracted_payload = Some(if config.include_metadata {
+ let record = DatabaseRecord {
+ table_name: config.table.to_string(),
+ operation_type: "SELECT".to_string(),
+ timestamp: Utc::now(),
+ data: serde_json::Value::Object(data),
+ old_data: None,
+ };
+ simd_json::to_vec(&record).map_err(|_| Error::InvalidRecord)?
+ } else {
+ simd_json::to_vec(&data).map_err(|_| Error::InvalidRecord)?
+ });
+ }
+
+ // Both paths above always assign extracted_payload before reaching here.
+ Ok(build_processed_row(
+ extracted_payload.unwrap(),
+ max_offset,
+ row_pk,
+ ))
+ }
+}
+
+fn extract_column_value(row: &MySqlRow, column_index: usize) -> Result<serde_json::Value, Error> {
+ let column = &row.columns()[column_index];
+ let type_name = column.type_info().name();
+
+ match type_name {
+ "BOOLEAN" => {
+ let value: Option<bool> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::Bool)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT" => {
+ let value: Option<i8> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT" => {
+ let value: Option<i16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT" | "INT" => {
+ let value: Option<i32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT" => {
+ let value: Option<i64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT UNSIGNED" => {
+ let value: Option<u8> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT UNSIGNED" => {
+ let value: Option<u16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT UNSIGNED" | "INT UNSIGNED" => {
+ let value: Option<u32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT UNSIGNED" => {
+ let value: Option<u64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "FLOAT" => {
+ let value: Option<f32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as f64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DOUBLE" => {
+ let value: Option<f64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DECIMAL" => {
+ let value: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIT" => {
+ let value: Option<u64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "YEAR" => {
+ let value: Option<u16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATE" => {
+ let value: Option<NaiveDate> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|d| serde_json::Value::String(d.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TIME" => {
+ let value: Option<NaiveTime> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|t| serde_json::Value::String(t.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATETIME" | "TIMESTAMP" => {
+ let value: Option<NaiveDateTime> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|dt| serde_json::Value::String(dt.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT" | "MEDIUMTEXT" | "LONGTEXT" | "ENUM" | "SET" => {
+ let value: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BINARY" | "VARBINARY" | "TINYBLOB" | "BLOB" | "MEDIUMBLOB" | "LONGBLOB" | "GEOMETRY" => {
+ let value: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|bytes| {
+ serde_json::Value::String(
+ base64::engine::general_purpose::STANDARD.encode(&bytes),
+ )
+ })
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "JSON" => {
+ let value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value.unwrap_or(serde_json::Value::Null))
+ }
+ "NULL" => Ok(serde_json::Value::Null),
+ _ => {
+ let column_name = column.name();
+ warn!(
+ "Column '{column_name}' has unrecognized MySQL type '{type_name}', \
+ attempting text extraction"
+ );
+ if let Ok(text) = row.try_get::<Option<String>, _>(column_index) {
+ return Ok(text
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null));
+ }
+ if let Ok(bytes) = row.try_get::<Option<Vec<u8>>, _>(column_index) {
+ return Ok(bytes
+ .map(|b| {
+ serde_json::Value::String(
+ base64::engine::general_purpose::STANDARD.encode(&b),
+ )
+ })
+ .unwrap_or(serde_json::Value::Null));
+ }
+ error!(
+ "Column '{column_name}' has unsupported MySQL type '{type_name}', \
+ returning null"
+ );
+ Ok(serde_json::Value::Null)
+ }
+ }
+}
+
+fn value_as_string(value: &serde_json::Value) -> Option<String> {
+ match value {
+ serde_json::Value::String(s) => Some(s.clone()),
+ serde_json::Value::Number(n) => Some(n.to_string()),
+ _ => None,
+ }
+}
+
+fn build_processed_row(
+ payload: Vec<u8>,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+) -> ProcessedRow {
+ let now = Utc::now().timestamp_millis() as u64;
+ ProcessedRow {
+ message: ProducedMessage {
+ id: Some(Uuid::new_v4().as_u128()),
+ headers: None,
+ checksum: None,
+ timestamp: Some(now),
+ origin_timestamp: Some(now),
+ payload,
+ },
+ max_offset,
+ row_pk,
+ }
+}
+
+fn to_snake_case(input: &str) -> String {
+ let mut result = String::new();
+ let mut prev_was_uppercase = false;
+ for (i, ch) in input.chars().enumerate() {
+ if ch.is_uppercase() {
+ if i > 0 && !prev_was_uppercase {
+ result.push('_');
+ }
+ if let Some(lc) = ch.to_lowercase().next() {
+ result.push(lc);
+ } else {
+ result.push(ch);
+ }
+ prev_was_uppercase = true;
+ } else {
+ result.push(ch);
+ prev_was_uppercase = false;
+ }
+ }
+ result
+}
+
+fn redact_connection_string(conn_str: &str) -> String {
+ if let Some(scheme_end) = conn_str.find("://") {
+ let scheme = &conn_str[..scheme_end + 3];
+ let rest = &conn_str[scheme_end + 3..];
+ let preview: String = rest.chars().take(3).collect();
+ return format!("{scheme}{preview}***");
+ }
+ let preview: String = conn_str.chars().take(3).collect();
+ format!("{preview}***")
+}
+
+fn quote_identifier(name: &str) -> Result<String, Error> {
+ if name.is_empty() {
+ return Err(Error::InvalidConfigValue(
+ "identifier must not be empty".to_string(),
+ ));
+ }
+ if name.contains('\0') {
+ return Err(Error::InvalidConfigValue(format!(
+ "identifier '{name}' contains NUL byte"
+ )));
+ }
+ let escaped = name.replace('`', "``");
+ Ok(format!("`{escaped}`"))
+}
+
+fn quote_qualified_identifier(name: &str) -> Result<String, Error> {
+ if !name.contains('.') {
+ return quote_identifier(name);
+ }
+ let parts: Result<Vec<_>, _> = name.split('.').map(quote_identifier).collect();
+ Ok(parts?.join("."))
+}
+
+fn format_offset_value(value: &str) -> String {
+ if value.parse::<i64>().is_ok() || value.parse::<f64>().is_ok_and(|v| v.is_finite()) {
+ value.to_string()
+ } else {
+ let escaped = value
+ .replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "");
+ format!("'{escaped}'")
+ }
+}
+
+fn is_transient_error(e: &sqlx::Error) -> bool {
+ match e {
+ sqlx::Error::Io(_) => true,
+ sqlx::Error::PoolTimedOut => true,
+ sqlx::Error::PoolClosed => false,
+ sqlx::Error::Protocol(_) => false,
+ // MySQL surfaces a numeric error code (e.g. 1213) and a SQLSTATE (e.g. "40001").
+ // `DatabaseError::code()` returns the SQLSTATE, so matching it against MySQL error
+ // numbers never fires. Downcast to the driver error and compare `number()` instead.
+ sqlx::Error::Database(db_err) => db_err
+ .try_downcast_ref::<MySqlDatabaseError>()
+ .is_some_and(|mysql_err| {
+ matches!(
+ mysql_err.number(),
+ // concurrency
+ 1213 | 1205 |
+ // server unavailability
+ 1053 | 1152 | 1080 |
+ // connection/network
+ 2006 | 2013 | 1158 | 1159 | 1160 | 1161 |
+ // resource exhaustion
+ 1040 | 1041
+ )
+ }),
+ _ => false,
+ }
+}
+
+async fn with_retry<T, F, Fut>(operation: F, max_retries: u32, delay_ms: u64) -> Result<T, Error>
+where
+ F: Fn() -> Fut,
+ Fut: std::future::Future<Output = Result<T, sqlx::Error>>,
+{
+ let mut attempts = 0;
+ loop {
+ match operation().await {
+ Ok(result) => return Ok(result),
+ Err(e) => {
+ attempts += 1;
+ if attempts >= max_retries || !is_transient_error(&e) {
Review Comment:
with_retry permanent/exhausted DB failure returns Error::InvalidRecord. Connection pool exhaustion (errno 1040) surfaces as row decode error. **Fix**: map infra failures to Error::Connection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r4187441046
##########
core/connectors/sources/mysql_source/Cargo.toml:
##########
@@ -0,0 +1,57 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[package]
+name = "iggy_connector_mysql_source"
+version = "0.1.0"
+description = "Iggy MySQL source connector supporting table polling for message streaming platform"
+edition = "2024"
+license = "Apache-2.0"
+keywords = ["iggy", "messaging", "streaming", "mysql", "polling"]
+categories = ["command-line-utilities", "database", "network-programming"]
+homepage = "https://iggy.apache.org"
+documentation = "https://iggy.apache.org/docs"
+repository = "https://github.com/apache/iggy"
+readme = "README.md"
+publish = false
+
+[package.metadata.cargo-machete]
+ignored = ["dashmap"]
+
+[lib]
+crate-type = ["cdylib", "lib"]
+
+[dependencies]
+async-trait = { workspace = true }
+base64 = { workspace = true }
+chrono = { workspace = true }
+dashmap = { workspace = true }
+humantime = { workspace = true }
+iggy_common = { workspace = true }
+iggy_connector_sdk = { workspace = true }
+rust_decimal = "=1.42.1"
Review Comment:
Fixed by the Ack staging. Delete/mark no longer runs in poll(). It runs in on_batch_result after Ack. A failed cleanup stays pending, and while anything is pending poll() always takes the pacing sleep and returns an empty batch without running the SELECT. So a persistent cleanup failure no longer re-reads or decodes rows. After 3 consecutive cleanup failures the connector stops, and the staged cleanup stays in the saved state to run again on restart.
also fixed the rust_decimal Fixed. rust_decimal is now declared in [workspace.dependencies] and the crate uses { workspace = true }, so mysql_source has no non-workspace dependencies left.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r4187552639
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.connect().await?;
+ self.validate_tracking_column().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ // Skip the pacing delay while draining a backlog
+ if !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ // Idle cycles produce no messages and leave offsets/processed_rows
+ // untouched, so return None to let the runtime skip the state fsync.
+ let persisted_state = if messages.is_empty() {
+ None
+ } else {
+ self.serialize_state(&state)
+ };
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn validate_tracking_column(&self) -> Result<(), Error> {
+ if self.config.custom_query.is_some() {
+ return Ok(());
+ }
+ let pool = self.get_pool()?;
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ continue;
+ }
+
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ if data_type.eq_ignore_ascii_case("json")
+ || column_type.eq_ignore_ascii_case("tinyint(1)")
+ {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type '{column_type}', \
+ which yields no ordered scalar cursor; use an integer, timestamp, or string column."
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+ let offset = format_offset_value(&offset_value);
+ let now = Utc::now();
+
+ let q = query
+ .replace("$table", "e_qualified_identifier(table)?)
+ .replace("$offset", &offset)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now_unix", &now.timestamp().to_string())
+ .replace("$now", &now.to_rfc3339());
+
+ Ok(q)
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[PkValue],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({placeholders})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ // The query and its binds are rebuilt on every attempt because with_retry
+ // calls the closure once per try and each `execute` consumes the query.
+ with_retry(
+ || {
+ let mut prepared = sqlx::query(sqlx::AssertSqlSafe(query.as_str()));
+ for id in ids {
+ prepared = match id {
+ PkValue::Text(text) => prepared.bind(text),
+ PkValue::Bytes(bytes) => prepared.bind(bytes),
+ };
+ }
+ prepared.execute(pool)
+ },
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ // A full batch means the LIMIT was hit, so more rows are waiting.
+ any_batch_full |= batch.messages.len() as u32 >= batch_size;
+ batches.push(batch);
Review Comment:
Fixed by the Ack staging. Delete/mark no longer runs in poll(). It runs in on_batch_result after Ack. A failed cleanup stays pending, and while anything is pending poll() always takes the pacing sleep and returns an empty batch without running the SELECT. So a persistent cleanup failure no longer re-reads or decodes rows. After 3 consecutive cleanup failures the connector stops, and the staged cleanup stays in the saved state to run again on restart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r4263325597
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,4071 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source,
+ source::SourceBatchResult, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+/// Upper bound on applying a batch's staged mark/delete after its Ack. The SDK waits
+/// `BATCH_RESULT_TIMEOUT` (30s) for the result, so this has to finish well inside it.
+const ACK_BATCH_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// MySQL rejects a prepared statement with more placeholders than this (error 1390), and
+/// the mark/delete binds one per row of the batch.
+const MAX_CLEANUP_BATCH_SIZE: u32 = 65_535;
+
+/// How many mark/delete statements in a row may fail before the connector stops. The
+/// operations stay in the persisted state and run again on restart.
+const MAX_CONSECUTIVE_CLEANUP_FAILURES: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ pending_batch: Mutex<Option<PendingBatch>>,
+ consecutive_cleanup_failures: AtomicU32,
+ verified_cleanup_keys: Mutex<HashSet<String>>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some("it has too few distinct values to be unique per row");
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some("binary values are compared as base64 text, which does not keep byte order");
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some("MySQL orders these by declaration index, not by label");
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL \
+ or not an ordered scalar (tinyint(1), JSON)"
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "row '{value}' is below the stored offset '{previous}' for tracking_column \
+ '{column}'. Filter the query with `{column} > $offset`"
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. Order the query ascending by '{column}'"
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+ /// Mark/delete work for a batch the runtime has already sent. It is persisted with
+ /// the batch's offsets so a crash between the send and the cleanup reruns it.
+ #[serde(default)]
+ pending_operations: Vec<PendingOperation>,
+}
+
+/// The rows of one table to mark or delete once their batch is acknowledged.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct PendingOperation {
+ table: String,
+ ids: Vec<PkValue>,
+ target: CleanupTarget,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct CleanupTarget {
+ key_column: String,
+ action: CleanupAction,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum CleanupAction {
+ Delete,
+ MarkProcessed { column: String },
+}
+
+/// The candidate state of the batch most recently handed to the runtime. Live state
+/// only takes it over on Ack, so a Nack leaves offsets and rows where they were.
+#[derive(Debug, Clone)]
+struct PendingBatch {
+ state: State,
+ /// The runtime has delivered the batch but its cleanup has not finished yet.
+ acknowledged: bool,
+ /// Cleanup finished and the next poll must persist `state` without the operations,
+ /// so a restart does not run them again.
+ retiring: bool,
+}
+
+struct PolledBatch {
+ messages: Vec<ProducedMessage>,
+ pending: Option<PendingBatch>,
+}
+
+enum CleanupKeyCheck {
+ Unique,
+ NotUnique,
+ TableMissing,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty, list the table even when custom_query hardcodes it"
+ .to_string(),
+ ));
+ }
+
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ if let Some(batch_size) = self.config.batch_size
+ && batch_size > MAX_CLEANUP_BATCH_SIZE
+ && self.cleanup_target().is_some()
+ {
+ return Err(Error::InitError(format!(
+ "batch_size {batch_size} exceeds {MAX_CLEANUP_BATCH_SIZE} with delete_after_read or \
+ processed_column set, MySQL allows at most {MAX_CLEANUP_BATCH_SIZE} placeholders \
+ in the statement that marks or deletes a batch"
+ )));
+ }
+
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "name the column custom_query orders by and filters with $offset"
+ } else {
+ "use a unique, monotonically increasing column"
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set, {guidance}"
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.validate_pending_cleanup()?;
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+ self.validate_cleanup_keys().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ if let Some(state) = self.retiring_state().await {
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: Some(self.serialize_pending_state(&state)?),
+ });
+ }
+
+ // Skip the pacing delay while draining a backlog, but never while a cleanup
+ // retry is pending, or a failing mark/delete would spin without a pause.
+ let cleanup_pending = self.pending_batch.lock().await.is_some();
+ if cleanup_pending || !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ {
+ let pending = self.pending_batch.lock().await;
+ if let Some(pending) = pending.as_ref() {
+ if !pending.acknowledged {
+ error!(
+ "{CONNECTOR_NAME} connector ID: {} was polled while a batch was still in flight",
+ self.id
+ );
+ return Err(Error::InvalidState);
+ }
+ // An empty batch whose Ack retries the cleanup the last one left behind.
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: None,
+ });
+ }
+ }
+
+ let polled = self.poll_tables().await?;
+
+ let processed_rows = match &polled.pending {
+ Some(pending) => pending.state.processed_rows,
+ None => self.state.lock().await.processed_rows,
+ };
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ }
+
+ // Idle cycles stage nothing, so return None to let the runtime skip the fsync.
+ let persisted_state = polled
+ .pending
+ .as_ref()
+ .map(|pending| self.serialize_pending_state(&pending.state))
+ .transpose()?;
+ *self.pending_batch.lock().await = polled.pending;
+
+ Ok(ProducedMessages {
+ schema,
+ messages: polled.messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> {
+ if result == SourceBatchResult::Nack {
+ // Dropping the candidate leaves the live offsets where they were, so the
+ // next poll reads the same rows again. Delivered batches keep their cleanup.
+ let mut pending = self.pending_batch.lock().await;
+ if pending
+ .as_ref()
+ .is_some_and(|pending| !pending.acknowledged && !pending.retiring)
+ {
+ pending.take();
+ }
+ return Ok(());
+ }
+
+ let Some(PendingBatch { mut state, .. }) = self.pending_batch.lock().await.clone() else {
+ return Ok(());
+ };
+
+ let operations = std::mem::take(&mut state.pending_operations);
+ let had_operations = !operations.is_empty();
+ let deadline = tokio::time::Instant::now() + ACK_BATCH_TIMEOUT;
+ let failed_operations = self.apply_pending_operations(operations, deadline).await?;
+
+ if failed_operations.is_empty() {
+ *self.state.lock().await = state.clone();
+ let mut pending = self.pending_batch.lock().await;
+ if had_operations {
+ *pending = Some(PendingBatch {
+ state,
+ acknowledged: false,
+ retiring: true,
+ });
+ } else {
+ pending.take();
+ }
+ } else {
+ state.pending_operations = failed_operations;
+ *self.pending_batch.lock().await = Some(PendingBatch {
+ state,
+ acknowledged: true,
+ retiring: false,
+ });
+ }
+
+ Ok(())
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ let state = restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ pending_operations: Vec::new(),
+ });
+ // The runtime only persists a batch's operations after sending it, so restored
+ // operations belong to a delivered batch and only the cleanup is left to do.
+ let pending_batch = (!state.pending_operations.is_empty()).then(|| PendingBatch {
+ state: state.clone(),
+ acknowledged: true,
+ retiring: false,
+ });
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(state),
+ pending_batch: Mutex::new(pending_batch),
+ consecutive_cleanup_failures: AtomicU32::new(0),
+ verified_cleanup_keys: Mutex::new(HashSet::new()),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ poisoned_tables: Mutex::new(HashSet::new()),
+ tracking_kinds: Mutex::new(HashMap::new()),
+ polls: AtomicU64::new(0),
+ cursor_regressions: Mutex::new(HashMap::new()),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ // `format_offset_value` escapes backslashes, which under NO_BACKSLASH_ESCAPES
+ // would reach MySQL as literal characters and filter on the wrong cursor.
+ .after_connect(|connection, _| {
+ Box::pin(async move {
+ sqlx::query(
+ "SET SESSION sql_mode = REPLACE(@@SESSION.sql_mode, 'NO_BACKSLASH_ESCAPES', '')",
+ )
+ .execute(&mut *connection)
+ .await?;
+ Ok(())
+ })
+ })
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn tracking_column(&self) -> &str {
+ self.config.tracking_column.as_deref().unwrap_or("id")
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn resolve_tracking_columns(&self) -> Result<HashMap<String, OffsetKind>, Error> {
+ // Whether the connector owns the query, and can therefore hold the schema to
+ // what that query will do with it.
+ let strict = self.config.custom_query.is_none();
+ let pool = self.get_pool()?;
+ let tracking_column = self.tracking_column();
+ let mut kinds = HashMap::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ }
+
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ }
+
+ warn!(
+ "tracking_column '{tracking_column}' is not a column of table '{table}'; \
+ assuming custom_query projects it. Its comparison order is resolved from \
+ the first result set instead."
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ // The column's type survives a plain projection, so it is worth rejecting
+ // whatever the query looks like. A custom_query aliasing an expression onto
+ // a name the table also uses is the one false positive, hence the hint.
+ if let Some(reason) = unusable_tracking_type(&data_type) {
+ let aliasing_hint = if self.config.custom_query.is_some() {
+ " If custom_query aliases a different expression to this name, rename the \
+ alias so it does not collide with the column."
+ } else {
+ ""
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': {reason}. Use an integer, timestamp, or string \
+ column.{aliasing_hint}"
+ )));
+ }
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; a row \
+ whose value is NULL cannot produce a cursor and will fail that table's batch. \
+ Have custom_query exclude those rows, or declare the column NOT NULL."
+ );
+ }
+
+ if is_collation_ordered(&data_type) {
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': values must be unique and increase in its collation order \
+ (zero-pad numbers, prefer a _bin collation)"
+ );
+ }
+
+ kinds.insert(table.clone(), offset_kind_for_data_type(&data_type));
+ }
+ Ok(kinds)
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn serialize_pending_state(&self, state: &State) -> Result<ConnectorState, Error> {
+ self.serialize_state(state).ok_or_else(|| {
+ Error::Serialization(format!("failed to serialize {CONNECTOR_NAME} state"))
+ })
+ }
+
+ async fn retiring_state(&self) -> Option<State> {
+ self.pending_batch
+ .lock()
+ .await
+ .as_ref()
+ .filter(|pending| pending.retiring)
+ .map(|pending| pending.state.clone())
+ }
+
+ fn primary_key_column(&self) -> &str {
+ self.config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(self.tracking_column())
+ }
+
+ fn cleanup_target(&self) -> Option<CleanupTarget> {
+ let action = if self.config.delete_after_read.unwrap_or(false) {
+ CleanupAction::Delete
+ } else if let Some(column) = &self.config.processed_column {
+ CleanupAction::MarkProcessed {
+ column: column.clone(),
+ }
+ } else {
+ return None;
+ };
+ Some(CleanupTarget {
+ key_column: self.primary_key_column().to_string(),
+ action,
+ })
+ }
+
+ /// Restored operations were staged under the config of the run that wrote them.
+ /// Running them under a different key or action would touch rows nobody read.
+ fn validate_pending_cleanup(&mut self) -> Result<(), Error> {
+ let configured = self.cleanup_target();
+ for operation in &self.state.get_mut().pending_operations {
+ if configured.as_ref() != Some(&operation.target) {
+ return Err(Error::InitError(format!(
+ "the cleanup configuration changed while cleanup of {} delivered row(s) in \
+ table '{}' is pending; restore primary_key_column '{}' and action {:?}",
+ operation.ids.len(),
+ operation.table,
+ operation.target.key_column,
+ operation.target.action
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ /// `WHERE key IN (...)` on a key that repeats marks or deletes every row sharing a
+ /// value, including ones that were never read. Tables that don't exist yet are
+ /// checked by the first poll that stages cleanup for them.
+ async fn validate_cleanup_keys(&mut self) -> Result<(), Error> {
+ let Some(target) = self.cleanup_target() else {
+ return Ok(());
+ };
+ let pool = self.get_pool()?;
+ let mut verified = HashSet::with_capacity(self.config.tables.len());
+ for table in &self.config.tables {
+ let check = cleanup_key_check(pool, table, &target.key_column)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to check primary_key_column '{}' on table '{table}': {e}",
+ target.key_column
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ verified.insert(table.clone());
+ }
+ CleanupKeyCheck::NotUnique => {
+ return Err(Error::InitError(not_unique_key_message(
+ table,
+ &target.key_column,
+ )));
+ }
+ CleanupKeyCheck::TableMissing => {}
+ }
+ }
+ *self.verified_cleanup_keys.get_mut() = verified;
+ Ok(())
+ }
+
+ async fn cleanup_key_is_unique(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ key_column: &str,
+ ) -> Result<bool, Error> {
+ if self.verified_cleanup_keys.lock().await.contains(table) {
+ return Ok(true);
+ }
+ let check = cleanup_key_check(pool, table, key_column)
+ .await
+ .map_err(|e| {
+ Error::Connection(format!(
+ "failed to check primary_key_column '{key_column}' on table '{table}': {e}"
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ self.verified_cleanup_keys
+ .lock()
+ .await
+ .insert(table.to_string());
+ Ok(true)
+ }
+ CleanupKeyCheck::NotUnique => Ok(false),
+ CleanupKeyCheck::TableMissing => Err(Error::Connection(format!(
+ "table '{table}' disappeared before its primary_key_column could be checked"
+ ))),
+ }
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ /// Best current knowledge of how a table's tracking values compare.
+ async fn tracking_kind(&self, table: &str) -> OffsetKind {
+ self.tracking_kinds
+ .lock()
+ .await
+ .get(table)
+ .copied()
+ .unwrap_or(OffsetKind::Unknown)
+ }
+
+ /// Whether the stored cursor is what limits the rows a poll can see. The built
+ /// polling query always filters on it; a `custom_query` does so only if it
+ /// interpolates `$offset`.
+ fn cursor_gates_query(&self) -> bool {
+ match &self.config.custom_query {
+ Some(query) => query.contains("$offset"),
+ None => true,
+ }
+ }
+
+ /// The offset a poll actually resumes from: the stored cursor, or the configured
+ /// starting point before anything has been stored.
+ fn effective_offset(&self, last_offset: &Option<String>) -> Option<String> {
+ last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let offset_value = self.effective_offset(last_offset).unwrap_or_default();
+ let offset = format_offset_value(&offset_value, kind);
+ let quoted_table = quote_qualified_identifier(table)?;
+ let limit = batch_size.to_string();
+ let now = Utc::now();
+ let now_unix = now.timestamp().to_string();
+ let now = now.to_rfc3339();
+ let replacements = [
+ ("$table", quoted_table.as_str()),
+ ("$offset", offset.as_str()),
+ ("$limit", limit.as_str()),
+ ("$now_unix", now_unix.as_str()),
+ ("$now", now.as_str()),
+ ];
+
+ Ok(substitute_query_tokens(query, &replacements))
+ }
+
+ /// The query is substituted as text and re-executed once per configured table on
+ /// every poll for the life of the process, so a shape that cannot terminate does
+ /// not fail, it repeats. What can be judged from the text alone is judged here,
+ /// at startup, rather than left to the first poll.
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let tracking_column = self.tracking_column();
+
+ match leading_statement_token(query) {
+ None => {
+ return Err(Error::InitError(
+ "custom_query has no statement: it is empty, or nothing but comments"
+ .to_string(),
+ ));
+ }
+ Some(token) if !READ_STATEMENT_STARTS.contains(&token.as_str()) => {
+ return Err(Error::InitError(format!(
+ "custom_query must be a read starting with SELECT, WITH or TABLE, found \
+ '{token}'"
+ )));
+ }
+ Some(_) => {}
+ }
+
+ // Collapsed so `ORDER\nBY` and `ORDER BY` read the same as `ORDER BY`; the
+ // check hard-fails, so a formatting difference must not decide it.
+ let normalized = query
+ .split_whitespace()
+ .collect::<Vec<_>>()
+ .join(" ")
+ .to_uppercase();
+
+ if query.contains("$offset") {
+ if !normalized.contains("ORDER BY") {
+ return Err(Error::InitError(format!(
+ "custom_query filters on $offset but has no ORDER BY, add \
+ `ORDER BY {tracking_column}`"
+ )));
+ }
+ } else if !self.config.delete_after_read.unwrap_or(false)
+ && self.config.processed_column.is_none()
+ {
+ return Err(Error::InitError(format!(
+ "custom_query must filter with $offset (`WHERE {tracking_column} > $offset`), or \
+ set delete_after_read or processed_column"
+ )));
+ }
+
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "custom_query uses $table but no tables are configured".to_string(),
+ ));
+ }
+ // Without $table the same SQL is executed once per table, so identical rows
+ // would be emitted repeatedly, each carrying a different table's metadata,
+ // message ID, and tracking offset. Table identity is not recoverable from a
+ // static query, so reject the combination instead of guessing one.
+ if !query.contains("$table") && self.config.tables.len() > 1 {
+ return Err(Error::InitError(format!(
+ "custom_query has no $table placeholder but {} tables are configured ({}), use \
+ $table in the query or configure a single table",
+ self.config.tables.len(),
+ self.config.tables.join(", ")
+ )));
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = self.effective_offset(last_offset) {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(&offset, kind)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn apply_pending_operations(
+ &self,
+ operations: Vec<PendingOperation>,
+ deadline: tokio::time::Instant,
+ ) -> Result<Vec<PendingOperation>, Error> {
+ let mut failed_operations = Vec::new();
+ let mut operations = operations.into_iter();
+ while let Some(operation) = operations.next() {
+ if tokio::time::Instant::now() >= deadline {
+ failed_operations.push(operation);
+ failed_operations.extend(operations);
+ break;
+ }
+ let result = tokio::time::timeout_at(deadline, async {
+ let pool = self.get_pool()?;
+ self.mark_or_delete_processed_rows(pool, &operation).await
+ })
+ .await;
+ match result {
+ Ok(Ok(())) => {
+ self.consecutive_cleanup_failures
+ .store(0, Ordering::Relaxed);
+ }
+ Ok(Err(error)) => {
+ self.record_cleanup_failure(&operation, &error)?;
+ failed_operations.push(operation);
+ }
+ Err(_) => {
+ self.record_cleanup_failure(
+ &operation,
+ "cleanup exceeded the 10s budget for one Ack",
+ )?;
+ failed_operations.push(operation);
+ }
+ }
+ }
+ Ok(failed_operations)
+ }
+
+ fn record_cleanup_failure(
+ &self,
+ operation: &PendingOperation,
+ error: impl std::fmt::Display,
+ ) -> Result<(), Error> {
+ let consecutive_failures = self
+ .consecutive_cleanup_failures
+ .fetch_add(1, Ordering::Relaxed)
+ .saturating_add(1);
+ error!(
+ "Failed to mark or delete {} delivered row(s) in table '{}' for {CONNECTOR_NAME} \
+ connector ID: {}, retrying on the next poll ({consecutive_failures} consecutive \
+ failures): {error}",
+ operation.ids.len(),
+ operation.table,
+ self.id
+ );
+ if consecutive_failures >= MAX_CONSECUTIVE_CLEANUP_FAILURES {
+ return Err(Error::Connection(format!(
+ "stopping {CONNECTOR_NAME} connector ID {} after {consecutive_failures} \
+ consecutive mark/delete failures; the cleanup stays in the saved state and \
+ runs again on restart",
+ self.id
+ )));
+ }
+ Ok(())
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ operation: &PendingOperation,
+ ) -> Result<(), Error> {
+ let PendingOperation { table, ids, target } = operation;
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(&target.key_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = match &target.action {
+ CleanupAction::Delete => {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
+ }
+ CleanupAction::MarkProcessed { column } => {
+ let quoted_processed = quote_identifier(column)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({placeholders})"
+ )
+ }
+ };
+
+ // The query and its binds are rebuilt on every attempt because with_retry
+ // calls the closure once per try and each `execute` consumes the query.
+ with_retry(
+ || {
+ let mut prepared = sqlx::query(sqlx::AssertSqlSafe(query.as_str()));
+ for id in ids {
+ prepared = match id {
+ PkValue::Text(text) => prepared.bind(text),
+ PkValue::Bytes(bytes) => prepared.bind(bytes),
+ };
+ }
+ prepared.execute(pool)
+ },
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<PolledBatch, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.tracking_column();
+ let pk_column = self.primary_key_column();
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ //
+ // An ordering violation is handled one step further. Retrying it would
+ // re-run the same query, re-decode the same rows, and hit the same failure
+ // forever, so the table is latched off instead of being retried each cycle.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ let poll_number = self.polls.fetch_add(1, Ordering::Relaxed) + 1;
+ let poisoned = { self.poisoned_tables.lock().await.clone() };
+ let mut newly_poisoned: Vec<String> = Vec::new();
+
+ for table in &self.config.tables {
+ if poisoned.contains(table) {
+ continue;
+ }
+
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ if !batch.messages.is_empty() {
+ self.cursor_regressions.lock().await.remove(table);
+ }
+ // A full batch means the LIMIT was hit, so more rows are waiting.
+ any_batch_full |= batch.messages.len() as u32 >= batch_size;
+ batches.push(batch);
+ }
+ Err(FetchError::Ordering(reason) | FetchError::MissingTrackingColumn(reason)) => {
+ error!(
+ "Table '{table}' at offset {}: {reason}. Table disabled until restart.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ newly_poisoned.push(table.clone());
+ }
+ Err(FetchError::CursorRegression(reason)) => {
+ // Cleared only by a batch that carried rows, so the run counts polls
+ // since the table last published anything. An idle poll produces as
+ // little as a rejected one does and must not read as recovery.
+ let consecutive = {
+ let mut regressions = self.cursor_regressions.lock().await;
+ let consecutive = regressions.entry(table.clone()).or_insert(0);
+ *consecutive += 1;
+ *consecutive
+ };
+ let offset = last_offset.as_deref().unwrap_or("<start>");
+ if consecutive >= CURSOR_REGRESSION_ERROR_THRESHOLD {
+ error!(
+ "Table '{table}' at offset {offset}: {reason}. Skipping this poll \
+ ({consecutive} consecutive polls)."
+ );
+ } else {
+ warn!("Table '{table}' at offset {offset}: {reason}. Skipping this poll.");
+ }
+ }
+ Err(FetchError::StaleOffsetKind(reason)) => {
+ warn!(
+ "Table '{table}' at offset {}: {reason}. Retrying on the next poll.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::UnusableTrackingValue(reason)) => {
+ error!(
+ "Table '{table}' at offset {}: {reason}. Retrying each poll until the row \
+ is fixed.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::Other(error)) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ if !newly_poisoned.is_empty() {
+ let mut poisoned_tables = self.poisoned_tables.lock().await;
+ poisoned_tables.extend(newly_poisoned);
+ if self
+ .config
+ .tables
+ .iter()
+ .all(|table| poisoned_tables.contains(table))
+ {
+ error!(
+ "All configured tables are disabled because their batches cannot yield a \
+ cursor; {CONNECTOR_NAME} connector with ID: {} is idle until restarted",
+ self.id
+ );
+ }
+ }
+
+ // From the next poll on a disabled table is skipped before anything is logged,
+ // so the one line written when it was disabled can scroll out of the operator's
+ // log window and leave a dark connector looking healthy.
+ if poll_number.is_multiple_of(DISABLED_TABLES_LOG_INTERVAL) {
+ let poisoned_tables = self.poisoned_tables.lock().await;
+ if !poisoned_tables.is_empty() {
+ let mut disabled: Vec<&str> = poisoned_tables.iter().map(String::as_str).collect();
+ disabled.sort_unstable();
+ error!(
+ "{} of {} configured table(s) are still disabled and are not being polled by \
+ {CONNECTOR_NAME} connector with ID: {}, and will stay that way until it is \
+ restarted: {}",
+ disabled.len(),
+ self.config.tables.len(),
+ self.id,
+ disabled.join(", ")
+ );
+ }
+ }
+
+ // Phase 2: stage each table's batch. Nothing touches MySQL or the live state
+ // here: the offsets and the mark/delete are applied in `on_batch_result` once
+ // the runtime has sent the batch and saved this candidate state, and a Nack
+ // throws the candidate away so the same rows are read again.
+ let cleanup_target = self.cleanup_target();
+ let mut candidate_state = self.state.lock().await.clone();
+ let mut operations = Vec::new();
+ let mut messages = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if let Some(target) = &cleanup_target
+ && !batch.messages.is_empty()
+ {
+ if batch.processed_ids.is_empty() {
+ error!(
+ "Table '{}': mark/delete is configured but no primary keys were extracted \
+ from {} row(s), skipping this cycle so rows are not published without \
+ being marked or deleted (check that '{pk_column}' is projected and scalar)",
+ batch.table,
+ batch.messages.len()
+ );
+ continue;
+ }
+
+ match self
+ .cleanup_key_is_unique(pool, &batch.table, &target.key_column)
+ .await
+ {
+ Ok(true) => {}
+ Ok(false) => {
+ error!(
+ "{}. Table disabled until restart.",
+ not_unique_key_message(&batch.table, &target.key_column)
+ );
+ self.poisoned_tables.lock().await.insert(batch.table);
+ continue;
+ }
+ Err(error) => {
+ error!("Table '{}' skipped this cycle: {error}", batch.table);
+ continue;
+ }
+ }
+
+ operations.push(PendingOperation {
+ table: batch.table.clone(),
+ ids: std::mem::take(&mut batch.processed_ids),
+ target: target.clone(),
+ });
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ candidate_state.tracking_offsets.insert(batch.table, offset);
+ }
+ }
+
+ let pending = (total_processed > 0).then(|| {
+ candidate_state.processed_rows += total_processed;
+ candidate_state.pending_operations = operations;
+ PendingBatch {
+ state: candidate_state,
+ acknowledged: false,
+ retiring: false,
+ }
+ });
+
+ self.last_batch_full
+ .store(any_batch_full, Ordering::Relaxed);
+ Ok(PolledBatch { messages, pending })
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, FetchError> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query_kind = self.tracking_kind(table).await;
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.substitute_query_params(custom_query, table, last_offset, batch_size, query_kind)?
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size, query_kind)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ // Settled once for the whole batch rather than per row, since the column list
+ // is a property of the result set.
+ if let Some(first) = rows.first() {
+ ensure_tracking_column_projected(
+ first.columns().iter().map(|column| column.name()),
+ tracking_column,
+ )?;
+ }
+
+ let resume_from = if self.cursor_gates_query() {
+ self.effective_offset(last_offset)
+ } else {
+ None
+ };
+
+ // The result set describes what this query actually returned, so it outranks
+ // whatever `information_schema` said at startup and is the only answer
+ // available for a computed column.
+ let kind = match rows.first().and_then(|row| {
+ observed_tracking_kind(row, tracking_column).filter(|observed| *observed != query_kind)
+ }) {
+ Some(observed) => {
+ self.tracking_kinds
+ .lock()
+ .await
+ .insert(table.to_string(), observed);
+ // The query was already sent with the stale kind, so its offset literal
+ // may be quoted for the wrong type.
+ if let Some(offset) = &resume_from
+ && offset_literal_diverged(offset, query_kind, observed)
+ {
+ return Err(FetchError::StaleOffsetKind(format!(
+ "tracking_column '{tracking_column}' type differs from the one detected \
+ at startup (filtered with {}, expected {}), batch discarded",
+ format_offset_value(offset, query_kind),
+ format_offset_value(offset, observed)
+ )));
+ }
+ observed
+ }
+ None => query_kind,
+ };
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ // Every row of a result set shares its columns, so the published key names are
+ // derived once per batch instead of once per row.
+ let column_names: Vec<String> = rows
+ .first()
+ .map(|row| {
+ row.columns()
+ .iter()
+ .map(|column| {
+ if table_config.snake_case_columns {
+ to_snake_case(column.name())
+ } else {
+ column.name().to_string()
+ }
+ })
+ .collect()
+ })
+ .unwrap_or_default();
+ let mut ordering = OrderingGuard::new(tracking_column, kind, resume_from);
+ for row in rows {
+ let processed = self
+ .process_row(&row, &table_config, &column_names)
+ .map_err(|e| {
+ error!(
+ "Failed to decode row in table '{table}' (pk {}): {e}",
+ pk_for_log(&row, table_config.pk_column)
+ );
+ e
+ })?;
+
+ let Some(value) = processed.tracking_value else {
+ return Err(unusable_tracking_value(
+ tracking_column,
+ &pk_for_log(&row, table_config.pk_column),
+ ));
+ };
+ ordering
+ .accept(value)
+ .map_err(|violation| match violation {
+ OrderingViolation::WithinBatch(reason) => FetchError::Ordering(reason),
+ OrderingViolation::BelowCursor(reason) => FetchError::CursorRegression(reason),
+ })?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+
+ batch.messages.push(processed.message);
+ }
+ batch.max_offset = ordering.into_cursor();
+
+ Ok(batch)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ column_names: &[String],
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<PkValue> = None;
+ let mut tracking_value: Option<String> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
+ let mut extracted_payload: Option<Vec<u8>> = None;
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ if name.eq_ignore_ascii_case(config.payload_col) {
+ extracted_payload =
+ Some(self.extract_payload_column(row, i, config.payload_format)?);
+ }
+ let is_tracking = name.eq_ignore_ascii_case(config.tracking_column);
+ let is_pk = name.eq_ignore_ascii_case(config.pk_column);
+ if is_tracking {
+ let value = extract_column_value(row, i)?;
+ tracking_value = column_value_as_string(row, i, &value)?;
+ if is_pk {
+ row_pk = pk_value_from_decoded(row, i, &value)?;
+ }
+ } else if is_pk {
+ row_pk = extract_pk_value(row, i)?;
+ }
+ }
+
+ // Every extract_payload_column arm yields Ok for a NULL column, so None
+ // here means the column was absent. Falling back to the whole-row JSON
+ // would publish bytes contradicting the schema derived from payload_format.
+ let payload = extracted_payload.ok_or_else(|| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': payload_column '{}' not present in the result set",
+ config.table, config.payload_col
+ ))
+ })?;
+
+ return Ok(build_processed_row(
+ config.table,
+ payload,
+ tracking_value,
+ row_pk,
+ ));
+ }
+
+ let mut data = serde_json::Map::new();
+ for ((i, column), column_name) in row.columns().iter().enumerate().zip(column_names) {
+ let name = column.name();
+ let value = extract_column_value(row, i)?;
+ if name.eq_ignore_ascii_case(config.tracking_column) {
+ tracking_value = column_value_as_string(row, i, &value)?;
+ }
+ if name.eq_ignore_ascii_case(config.pk_column) {
+ row_pk = pk_value_from_decoded(row, i, &value)?;
+ }
+ data.insert(column_name.clone(), value);
+ }
+
+ let payload = if config.include_metadata {
+ let record = DatabaseRecord {
+ table_name: config.table.to_string(),
+ operation_type: "SELECT".to_string(),
+ timestamp: Utc::now(),
+ data: serde_json::Value::Object(data),
+ old_data: None,
+ };
+ simd_json::to_vec(&record).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': failed to serialize row to JSON: {e}",
+ config.table
+ ))
+ })?
+ } else {
+ simd_json::to_vec(&data).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': failed to serialize row to JSON: {e}",
+ config.table
+ ))
+ })?
+ };
+
+ Ok(build_processed_row(
+ config.table,
+ payload,
+ tracking_value,
+ row_pk,
+ ))
+ }
+}
+
+/// Type-faithful primary-key extraction. Binary columns are taken as raw bytes
+/// rather than routed through the base64 path in `extract_column_value`, so the
+/// value can be bound back into `WHERE pk IN (...)` and match the row. Everything
+/// else takes the string form MySQL parses back to the same value. Returns None
+/// only for a NULL pk, which a real primary key cannot be.
+fn extract_pk_value(row: &MySqlRow, column_index: usize) -> Result<Option<PkValue>, Error> {
+ if is_binary_column(row, column_index) {
+ return binary_pk_value(row, column_index);
+ }
+ let value = extract_column_value(row, column_index)?;
+ pk_value_from_decoded(row, column_index, &value)
+}
+
+/// Like `extract_pk_value` for a column the caller has already decoded, so the
+/// common key column is not decoded a second time per row.
+fn pk_value_from_decoded(
+ row: &MySqlRow,
+ column_index: usize,
+ value: &serde_json::Value,
+) -> Result<Option<PkValue>, Error> {
+ if is_binary_column(row, column_index) {
+ return binary_pk_value(row, column_index);
+ }
+ Ok(column_value_as_string(row, column_index, value)?.map(PkValue::Text))
+}
+
+fn is_binary_column(row: &MySqlRow, column_index: usize) -> bool {
+ matches!(
+ row.columns()[column_index].type_info().name(),
+ "BINARY" | "VARBINARY" | "TINYBLOB" | "BLOB" | "MEDIUMBLOB" | "LONGBLOB"
+ )
+}
+
+fn binary_pk_value(row: &MySqlRow, column_index: usize) -> Result<Option<PkValue>, Error> {
+ let column = &row.columns()[column_index];
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "primary key column '{}' (MySQL type '{}'): {e}",
+ column.name(),
+ column.type_info().name()
+ ))
+ })?;
+ Ok(bytes.map(PkValue::Bytes))
+}
+
+fn extract_column_value(row: &MySqlRow, column_index: usize) -> Result<serde_json::Value, Error> {
+ let column = &row.columns()[column_index];
+ let type_name = column.type_info().name();
+
+ let to_err = |e: sqlx::Error| {
+ Error::InvalidRecordValue(format!(
+ "column '{}' (MySQL type '{type_name}'): {e}",
+ column.name()
+ ))
+ };
+
+ match type_name {
+ "BOOLEAN" => {
+ let value: Option<bool> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::Bool)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT" => {
+ let value: Option<i8> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT" => {
+ let value: Option<i16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT" | "INT" => {
+ let value: Option<i32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT" => {
+ let value: Option<i64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT UNSIGNED" => {
+ let value: Option<u8> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT UNSIGNED" => {
+ let value: Option<u16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT UNSIGNED" | "INT UNSIGNED" => {
+ let value: Option<u32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT UNSIGNED" => {
+ let value: Option<u64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "FLOAT" => {
+ let value: Option<f32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as f64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DOUBLE" => {
+ let value: Option<f64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DECIMAL" => {
+ let value: Option<rust_decimal::Decimal> = row.try_get(column_index).map_err(to_err)?;
Review Comment:
fixed, DECIMAL is now decoded as BigDecimal and written with to_plain_string(), so payloads and the cursor keep every digit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on PR #3568:
URL: https://github.com/apache/iggy/pull/3568#issuecomment-5427585379
@ryerraguntla @hubcio can you pls reopen this pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3531965664
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).execute(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. A table's messages are emitted
+ // only after its rows are marked or deleted, so a failure here can never
+ // leave a table marked-but-unpublished. `mark_or_delete_processed_rows`
+ // retries transient errors; a permanent failure isolates that one table
+ // (its rows stay in MySQL and are retried next cycle) while the others
+ // still flow.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if !batch.processed_ids.is_empty()
+ && let Err(error) = self
+ .mark_or_delete_processed_rows(
+ pool,
+ &batch.table,
+ pk_column,
+ &batch.processed_ids,
+ )
+ .await
+ {
+ error!(
+ "Failed to mark or delete processed rows for table '{}', skipping this cycle: {error}",
+ batch.table
+ );
+ continue;
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ state_updates.push((batch.table, offset));
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
+ }
+ state.last_poll_time = Utc::now();
+ }
+
+ Ok(messages)
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, Error> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.validate_custom_query(custom_query)?;
+ self.substitute_query_params(custom_query, table, last_offset, batch_size)
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ for row in rows {
+ let processed = self.process_row(&row, &table_config)?;
Review Comment:
using per-row skip can cause dataloss as we are updating the offset to the last polled row, for now i have emphasised on making the error message more detailed so the operator can pin pointedly know the exact reason and fix it, documented the same in the readme.
As for the DLQ can take that as a fresh issue after going through the discussions around it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3623186779
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1563 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ // Skip the pacing delay while draining a backlog
+ if !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ // Idle cycles produce no messages and leave offsets/processed_rows
+ // untouched, so return None to let the runtime skip the state fsync.
+ let persisted_state = if messages.is_empty() {
+ None
+ } else {
+ self.serialize_state(&state)
+ };
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+ let offset = format_offset_value(&offset_value);
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now_unix", &now.timestamp().to_string())
+ .replace("$now", &now.to_rfc3339())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() || id.parse::<u64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).execute(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ // A full batch means the LIMIT was hit, so more rows are waiting.
+ any_batch_full |= batch.messages.len() as u32 >= batch_size;
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. If the mark/delete call fails
+ // for a table we just skip its messages this cycle instead of queuing them
+ // - `mark_or_delete_processed_rows` already retries transient errors, so a
+ // failure here is permanent and the table's rows stay in MySQL to be
+ // retried next cycle, without blocking the others.
+ //
+ // Doesn't cover the case where mark/delete succeeds but the runtime never
+ // gets to publish the batch (crash, send failure, etc) - those rows are
+ // gone from MySQL with nothing delivered. That's just how
+ // delete_after_read/processed_column work, not something this loop fixes.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ let mark_or_delete = self.config.delete_after_read.unwrap_or(false)
+ || self.config.processed_column.is_some();
+
+ for mut batch in batches {
+ if mark_or_delete && !batch.messages.is_empty() && batch.processed_ids.is_empty() {
+ warn!(
+ "Table '{}': mark/delete is configured but no primary keys were extracted \
+ from {} row(s); rows publish without being marked or deleted \
+ (check that '{pk_column}' is projected and scalar)",
+ batch.table,
+ batch.messages.len()
+ );
+ }
+
+ if !batch.processed_ids.is_empty()
+ && let Err(error) = self
+ .mark_or_delete_processed_rows(
+ pool,
+ &batch.table,
+ pk_column,
+ &batch.processed_ids,
+ )
+ .await
+ {
+ error!(
+ "Failed to mark or delete processed rows for table '{}', skipping this cycle: {error}",
+ batch.table
+ );
+ continue;
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ state_updates.push((batch.table, offset));
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
+ }
+ }
+
+ self.last_batch_full
+ .store(any_batch_full, Ordering::Relaxed);
+ Ok(messages)
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, Error> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.substitute_query_params(custom_query, table, last_offset, batch_size)
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ for row in rows {
+ let processed = self.process_row(&row, &table_config).map_err(|e| {
+ error!(
+ "Failed to decode row in table '{table}' (pk {}): {e}",
+ pk_for_log(&row, table_config.pk_column)
+ );
+ e
+ })?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+ if let Some(offset) = processed.max_offset {
+ batch.max_offset = Some(offset);
+ }
+
+ batch.messages.push(processed.message);
+ }
+
+ Ok(batch)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<String> = None;
+ let mut max_offset: Option<String> = None;
+ let mut extracted_payload: Option<Vec<u8>> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ if name == config.payload_col {
+ extracted_payload =
+ Some(self.extract_payload_column(row, i, config.payload_format)?);
+ }
+ if name == config.tracking_column || name == config.pk_column {
+ let as_string = value_as_string(&extract_column_value(row, i)?);
Review Comment:
added open-time schema probe for tracking_column, PK already is DB-enforced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3625193203
##########
core/connectors/sources/mysql_source/Cargo.toml:
##########
@@ -0,0 +1,54 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[package]
+name = "iggy_connector_mysql_source"
+version = "0.1.0"
+description = "Iggy MySQL source connector supporting table polling for message streaming platform"
+edition = "2024"
+license = "Apache-2.0"
+keywords = ["iggy", "messaging", "streaming", "mysql", "polling"]
+categories = ["command-line-utilities", "database", "network-programming"]
+homepage = "https://iggy.apache.org"
+documentation = "https://iggy.apache.org/docs"
+repository = "https://github.com/apache/iggy"
+readme = "README.md"
+publish = false
+
+[package.metadata.cargo-machete]
+ignored = ["dashmap"]
+
+[lib]
+crate-type = ["cdylib", "lib"]
+
+[dependencies]
+async-trait = { workspace = true }
+base64 = { workspace = true }
+chrono = { workspace = true }
Review Comment:
dashmap is required by source_connector! macro expansion, cargo-machete can't see through macro expansion, thats why machete ignore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3479575148
##########
.gitignore:
##########
@@ -42,3 +42,5 @@ go.work
core/bench/dashboard/frontend/dist
LICENSE-binary
**/LICENSE-binary
+local_test_connectors
+local_test_runtime.toml
Review Comment:
i dont see this file being added anywhere in the test in this PR nor in iggy as a whole. please drop it and add it to your own local `.gitignore_global`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #3568:
URL: https://github.com/apache/iggy/pull/3568#issuecomment-4804795896
## [Codecov](https://app.codecov.io/gh/apache/iggy/pull/3568?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 73.24%. Comparing base ([`bc68827`](https://app.codecov.io/gh/apache/iggy/commit/bc68827aea2d844069974b6514d8f89ff0e07bf7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`1c4d3d9`](https://app.codecov.io/gh/apache/iggy/commit/1c4d3d9ae338ef4dfee0230450f38cae335b4aa5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #3568 +/- ##
============================================
- Coverage 74.02% 73.24% -0.78%
Complexity 937 937
============================================
Files 1247 1184 -63
Lines 127567 122880 -4687
Branches 103435 103348 -87
============================================
- Hits 94427 90008 -4419
+ Misses 30103 29835 -268
Partials 3037 3037
```
| [Components](https://app.codecov.io/gh/apache/iggy/pull/3568/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [Rust Core](https://app.codecov.io/gh/apache/iggy/pull/3568/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.67% <ø> (ø)` | |
| [Java SDK](https://app.codecov.io/gh/apache/iggy/pull/3568/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.44% <ø> (ø)` | |
| [C# SDK](https://app.codecov.io/gh/apache/iggy/pull/3568/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `72.10% <ø> (ø)` | |
| [Python SDK](https://app.codecov.io/gh/apache/iggy/pull/3568/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `88.88% <ø> (ø)` | |
| [PHP SDK](https://app.codecov.io/gh/apache/iggy/pull/3568/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `84.29% <ø> (ø)` | |
| [Node SDK](https://app.codecov.io/gh/apache/iggy/pull/3568/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `87.27% <ø> (-3.86%)` | :arrow_down: |
| [Go SDK](https://app.codecov.io/gh/apache/iggy/pull/3568/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `40.14% <ø> (ø)` | |
[see 81 files with indirect coverage changes](https://app.codecov.io/gh/apache/iggy/pull/3568/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on PR #3568:
URL: https://github.com/apache/iggy/pull/3568#issuecomment-5039235428
/ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on PR #3568:
URL: https://github.com/apache/iggy/pull/3568#issuecomment-4939749668
/ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3554059170
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).execute(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. A table's messages are emitted
+ // only after its rows are marked or deleted, so a failure here can never
+ // leave a table marked-but-unpublished. `mark_or_delete_processed_rows`
+ // retries transient errors; a permanent failure isolates that one table
+ // (its rows stay in MySQL and are retried next cycle) while the others
+ // still flow.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if !batch.processed_ids.is_empty()
+ && let Err(error) = self
+ .mark_or_delete_processed_rows(
+ pool,
+ &batch.table,
+ pk_column,
+ &batch.processed_ids,
+ )
+ .await
+ {
+ error!(
+ "Failed to mark or delete processed rows for table '{}', skipping this cycle: {error}",
+ batch.table
+ );
+ continue;
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ state_updates.push((batch.table, offset));
+ }
+ }
+
Review Comment:
poll_tables bumps in-memory tracking_offsets before return; disk state saved only on producer.send() OK. Live send fail → next poll reads advanced offset from memory → rows skipped forever. **Fix**: platform sibling issue (#2927/#2928/#2940); document in README.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio closed pull request #3568: feat(connectors): add MySQL source polling connector
URL: https://github.com/apache/iggy/pull/3568
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r4183250378
##########
core/connectors/sources/mysql_source/Cargo.toml:
##########
@@ -0,0 +1,57 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[package]
+name = "iggy_connector_mysql_source"
+version = "0.1.0"
+description = "Iggy MySQL source connector supporting table polling for message streaming platform"
+edition = "2024"
+license = "Apache-2.0"
+keywords = ["iggy", "messaging", "streaming", "mysql", "polling"]
+categories = ["command-line-utilities", "database", "network-programming"]
+homepage = "https://iggy.apache.org"
+documentation = "https://iggy.apache.org/docs"
+repository = "https://github.com/apache/iggy"
+readme = "README.md"
+publish = false
+
+[package.metadata.cargo-machete]
+ignored = ["dashmap"]
+
+[lib]
+crate-type = ["cdylib", "lib"]
+
+[dependencies]
+async-trait = { workspace = true }
+base64 = { workspace = true }
+chrono = { workspace = true }
+dashmap = { workspace = true }
+humantime = { workspace = true }
+iggy_common = { workspace = true }
+iggy_connector_sdk = { workspace = true }
+rust_decimal = "=1.42.1"
Review Comment:
rust_decimal = "=1.42.1" pinned as the crate's only non-workspace dependency, while the committed Cargo.lock has rust_decimal 1.43.0. ecosystem reports cargo metadata --locked exits 101 at PR head — CI's cargo build --locked would fail outright on this PR as committed. distsys independently flagged the same dependency as not using { workspace = true } despite the PR itself adding rust_decimal to the workspace's sqlx features. Fix: move to [workspace.dependencies], use { workspace = true }, regenerate the lockfile.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,3317 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some(
+ "at most 256 distinct values cannot be unique per row, and at display width 1 MySQL \
+ spells the type BOOLEAN, which reaches the connector as true/false with no ordered \
+ scalar form",
+ );
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some(
+ "binary values are carried as base64 text, which does not order the way the raw \
+ bytes do, and that text is what gets compared against the column",
+ );
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some(
+ "MySQL orders these by declaration index while only their labels reach the \
+ connector, and a bounded set of labels cannot be unique per row either",
+ );
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]. Every row's cursor \
+ comes from that column, so the same rows would be re-published on every poll while the \
+ stored offset never moved.",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL, or \
+ of a type with no ordered scalar form (tinyint(1), JSON). The cursor is taken from that \
+ value, so publishing the row would advance nothing and the same rows would be re-read on \
+ every poll."
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "a row arrived below the offset the poll resumed from by tracking_column \
+ '{column}': stored offset '{previous}', first row '{value}'. Taking the \
+ cursor from the last row would move it backwards. Filter the query on \
+ '{column}' with $offset so a batch cannot reach behind its own cursor."
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. The cursor is taken from the last row, so it would \
+ move backwards and rows would be skipped permanently. Order the query \
+ ascending by '{column}'."
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ // Every poll iterates `tables`, so an empty list makes the connector a
+ // silent no-op rather than an obvious misconfiguration.
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty. Even when custom_query hardcodes the table, list it \
+ here: the table name supplies the offset key, the deterministic message ID, \
+ and the table_name metadata field"
+ .to_string(),
+ ));
+ }
+
+ // A zero batch would make every fetch look like a full batch, so `poll`
+ // would drop its pacing sleep and spin on `LIMIT 0` queries forever.
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ // Only when the operator owns the ORDER BY. The built query sorts ascending by
+ // construction, so a one-row batch is safe there and legitimate for a slow table.
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset: a \
+ one-row batch offers no pair to compare, so the ordering check reduces to \
+ the row against the offset it resumed from, and a query ordered descending \
+ opens above that offset on every poll. The cursor would walk to the table's \
+ maximum and skip every row below it with nothing reporting it"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "the connector does not parse custom_query, so it has to be told which projected \
+ column supplies the cursor: name the one the query orders by and filters with \
+ $offset"
+ } else {
+ "it has to be unique and monotonically increasing under the order MySQL sorts it \
+ in."
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set: every row's cursor is taken from it, and {guidance}."
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ // Skip the pacing delay while draining a backlog
+ if !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ // Idle cycles produce no messages and leave offsets/processed_rows
+ // untouched, so return None to let the runtime skip the state fsync.
+ let persisted_state = if messages.is_empty() {
Review Comment:
serialize_state returning None silently becomes state: None; runtime logs "No state provided" and Acks anyway (runtime/source.rs:759-765). A serialization failure after rows are already deleted and the cursor already advanced = silent, unrecoverable loss with no error surfaced. postgres_source maps this to Err(Error::Serialization) instead (postgres lib.rs:318-320)
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,3317 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
Review Comment:
No #[serde(deny_unknown_fields)] on MySqlSourceConfig (lib.rs:75-96) — a typo'd field (e.g. delete_after_reads instead of delete_after_read) silently falls back to the default instead of erroring, in the worst case silently enabling a destructive DELETE path the operator never intended to turn on. Established repo convention per multiple other connectors.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,3317 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some(
+ "at most 256 distinct values cannot be unique per row, and at display width 1 MySQL \
+ spells the type BOOLEAN, which reaches the connector as true/false with no ordered \
+ scalar form",
+ );
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some(
+ "binary values are carried as base64 text, which does not order the way the raw \
+ bytes do, and that text is what gets compared against the column",
+ );
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some(
+ "MySQL orders these by declaration index while only their labels reach the \
+ connector, and a bounded set of labels cannot be unique per row either",
+ );
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]. Every row's cursor \
+ comes from that column, so the same rows would be re-published on every poll while the \
+ stored offset never moved.",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL, or \
+ of a type with no ordered scalar form (tinyint(1), JSON). The cursor is taken from that \
+ value, so publishing the row would advance nothing and the same rows would be re-read on \
+ every poll."
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "a row arrived below the offset the poll resumed from by tracking_column \
+ '{column}': stored offset '{previous}', first row '{value}'. Taking the \
+ cursor from the last row would move it backwards. Filter the query on \
+ '{column}' with $offset so a batch cannot reach behind its own cursor."
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. The cursor is taken from the last row, so it would \
+ move backwards and rows would be skipped permanently. Order the query \
+ ascending by '{column}'."
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ // Every poll iterates `tables`, so an empty list makes the connector a
+ // silent no-op rather than an obvious misconfiguration.
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty. Even when custom_query hardcodes the table, list it \
+ here: the table name supplies the offset key, the deterministic message ID, \
+ and the table_name metadata field"
+ .to_string(),
+ ));
+ }
+
+ // A zero batch would make every fetch look like a full batch, so `poll`
+ // would drop its pacing sleep and spin on `LIMIT 0` queries forever.
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ // Only when the operator owns the ORDER BY. The built query sorts ascending by
+ // construction, so a one-row batch is safe there and legitimate for a slow table.
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset: a \
+ one-row batch offers no pair to compare, so the ordering check reduces to \
+ the row against the offset it resumed from, and a query ordered descending \
+ opens above that offset on every poll. The cursor would walk to the table's \
+ maximum and skip every row below it with nothing reporting it"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "the connector does not parse custom_query, so it has to be told which projected \
+ column supplies the cursor: name the one the query orders by and filters with \
+ $offset"
+ } else {
+ "it has to be unique and monotonically increasing under the order MySQL sorts it \
+ in."
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set: every row's cursor is taken from it, and {guidance}."
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ // Skip the pacing delay while draining a backlog
+ if !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ // Idle cycles produce no messages and leave offsets/processed_rows
+ // untouched, so return None to let the runtime skip the state fsync.
+ let persisted_state = if messages.is_empty() {
+ None
+ } else {
+ self.serialize_state(&state)
+ };
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ poisoned_tables: Mutex::new(HashSet::new()),
+ tracking_kinds: Mutex::new(HashMap::new()),
+ polls: AtomicU64::new(0),
+ cursor_regressions: Mutex::new(HashMap::new()),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ /// The column every cursor is read from. `open()` refuses a config that leaves it
+ /// unset, so the fallback is unreachable in a running connector.
+ fn tracking_column(&self) -> &str {
+ self.config.tracking_column.as_deref().unwrap_or("id")
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn resolve_tracking_columns(&self) -> Result<HashMap<String, OffsetKind>, Error> {
+ // Whether the connector owns the query, and can therefore hold the schema to
+ // what that query will do with it.
+ let strict = self.config.custom_query.is_none();
+ let pool = self.get_pool()?;
+ let tracking_column = self.tracking_column();
+ let mut kinds = HashMap::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ }
+
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ }
+
+ warn!(
+ "tracking_column '{tracking_column}' is not a column of table '{table}'; \
+ assuming custom_query projects it. Its comparison order is resolved from \
+ the first result set instead."
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
Review Comment:
cursor-offset (state.tracking_offsets) committed unconditionally inside poll_tables, before the batch reaches Iggy or state is persisted. Hits every deployment, not just delete/mark-configured ones. A single failed send silently and permanently skips those rows on the next poll's WHERE tracking_column > offset, no crash needed. **Fix**: stage, commit only on Ack.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.connect().await?;
+ self.validate_tracking_column().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ // Skip the pacing delay while draining a backlog
+ if !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ // Idle cycles produce no messages and leave offsets/processed_rows
+ // untouched, so return None to let the runtime skip the state fsync.
+ let persisted_state = if messages.is_empty() {
+ None
+ } else {
+ self.serialize_state(&state)
+ };
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn validate_tracking_column(&self) -> Result<(), Error> {
+ if self.config.custom_query.is_some() {
+ return Ok(());
+ }
+ let pool = self.get_pool()?;
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ continue;
+ }
+
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ if data_type.eq_ignore_ascii_case("json")
+ || column_type.eq_ignore_ascii_case("tinyint(1)")
+ {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type '{column_type}', \
+ which yields no ordered scalar cursor; use an integer, timestamp, or string column."
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+ let offset = format_offset_value(&offset_value);
+ let now = Utc::now();
+
+ let q = query
+ .replace("$table", "e_qualified_identifier(table)?)
+ .replace("$offset", &offset)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now_unix", &now.timestamp().to_string())
+ .replace("$now", &now.to_rfc3339());
+
+ Ok(q)
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[PkValue],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({placeholders})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ // The query and its binds are rebuilt on every attempt because with_retry
+ // calls the closure once per try and each `execute` consumes the query.
+ with_retry(
+ || {
+ let mut prepared = sqlx::query(sqlx::AssertSqlSafe(query.as_str()));
+ for id in ids {
+ prepared = match id {
+ PkValue::Text(text) => prepared.bind(text),
+ PkValue::Bytes(bytes) => prepared.bind(bytes),
+ };
+ }
+ prepared.execute(pool)
+ },
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ // A full batch means the LIMIT was hit, so more rows are waiting.
+ any_batch_full |= batch.messages.len() as u32 >= batch_size;
+ batches.push(batch);
Review Comment:
any_batch_full is computed in phase 1 before phase 2's outcome is known; a persistent delete/mark failure on a full batch causes poll() to return 0 messages, Ack, skip the pacing sleep, and re-poll immediately — an unbounded tight loop doing a full SELECT + decode every iteration with zero backoff, hammering the database. Independently confirmed by distsys as a "hot-spin / DB DoS" finding via the same mechanism. Similarly at lines 1360+552-554.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,3317 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some(
+ "at most 256 distinct values cannot be unique per row, and at display width 1 MySQL \
+ spells the type BOOLEAN, which reaches the connector as true/false with no ordered \
+ scalar form",
+ );
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some(
+ "binary values are carried as base64 text, which does not order the way the raw \
+ bytes do, and that text is what gets compared against the column",
+ );
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some(
+ "MySQL orders these by declaration index while only their labels reach the \
+ connector, and a bounded set of labels cannot be unique per row either",
+ );
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]. Every row's cursor \
+ comes from that column, so the same rows would be re-published on every poll while the \
+ stored offset never moved.",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL, or \
+ of a type with no ordered scalar form (tinyint(1), JSON). The cursor is taken from that \
+ value, so publishing the row would advance nothing and the same rows would be re-read on \
+ every poll."
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "a row arrived below the offset the poll resumed from by tracking_column \
+ '{column}': stored offset '{previous}', first row '{value}'. Taking the \
+ cursor from the last row would move it backwards. Filter the query on \
+ '{column}' with $offset so a batch cannot reach behind its own cursor."
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. The cursor is taken from the last row, so it would \
+ move backwards and rows would be skipped permanently. Order the query \
+ ascending by '{column}'."
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ // Every poll iterates `tables`, so an empty list makes the connector a
+ // silent no-op rather than an obvious misconfiguration.
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty. Even when custom_query hardcodes the table, list it \
+ here: the table name supplies the offset key, the deterministic message ID, \
+ and the table_name metadata field"
+ .to_string(),
+ ));
+ }
+
+ // A zero batch would make every fetch look like a full batch, so `poll`
+ // would drop its pacing sleep and spin on `LIMIT 0` queries forever.
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ // Only when the operator owns the ORDER BY. The built query sorts ascending by
+ // construction, so a one-row batch is safe there and legitimate for a slow table.
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset: a \
+ one-row batch offers no pair to compare, so the ordering check reduces to \
+ the row against the offset it resumed from, and a query ordered descending \
+ opens above that offset on every poll. The cursor would walk to the table's \
+ maximum and skip every row below it with nothing reporting it"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "the connector does not parse custom_query, so it has to be told which projected \
+ column supplies the cursor: name the one the query orders by and filters with \
+ $offset"
+ } else {
+ "it has to be unique and monotonically increasing under the order MySQL sorts it \
+ in."
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set: every row's cursor is taken from it, and {guidance}."
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ // Skip the pacing delay while draining a backlog
+ if !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ // Idle cycles produce no messages and leave offsets/processed_rows
+ // untouched, so return None to let the runtime skip the state fsync.
+ let persisted_state = if messages.is_empty() {
+ None
+ } else {
+ self.serialize_state(&state)
+ };
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ poisoned_tables: Mutex::new(HashSet::new()),
+ tracking_kinds: Mutex::new(HashMap::new()),
+ polls: AtomicU64::new(0),
+ cursor_regressions: Mutex::new(HashMap::new()),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ /// The column every cursor is read from. `open()` refuses a config that leaves it
+ /// unset, so the fallback is unreachable in a running connector.
+ fn tracking_column(&self) -> &str {
+ self.config.tracking_column.as_deref().unwrap_or("id")
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn resolve_tracking_columns(&self) -> Result<HashMap<String, OffsetKind>, Error> {
+ // Whether the connector owns the query, and can therefore hold the schema to
+ // what that query will do with it.
+ let strict = self.config.custom_query.is_none();
+ let pool = self.get_pool()?;
+ let tracking_column = self.tracking_column();
+ let mut kinds = HashMap::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ }
+
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ }
+
+ warn!(
+ "tracking_column '{tracking_column}' is not a column of table '{table}'; \
+ assuming custom_query projects it. Its comparison order is resolved from \
+ the first result set instead."
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ // The column's type survives a plain projection, so it is worth rejecting
+ // whatever the query looks like. A custom_query aliasing an expression onto
+ // a name the table also uses is the one false positive, hence the hint.
+ if let Some(reason) = unusable_tracking_type(&data_type) {
+ let aliasing_hint = if self.config.custom_query.is_some() {
+ " If custom_query aliases a different expression to this name, rename the \
+ alias so it does not collide with the column."
+ } else {
+ ""
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': {reason}. Use an integer, timestamp, or string \
+ column.{aliasing_hint}"
+ )));
+ }
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; a row \
+ whose value is NULL cannot produce a cursor and will fail that table's batch. \
+ Have custom_query exclude those rows, or declare the column NOT NULL."
+ );
+ }
+
+ if is_collation_ordered(&data_type) {
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}', so its cursor is compared in the column's collation and has \
+ to be unique and monotonically increasing in that order. Unpadded digit \
+ strings are not: '17' sorts below '9', after which the connector idles at \
+ the collation maximum forever. Zero-padded numbers, ULIDs and KSUIDs are. A \
+ case- or accent-insensitive collation such as the default utf8mb4_0900_ai_ci \
+ also breaks uniqueness, since 'ORD-1a' equals 'ORD-1A' and one of the two is \
+ dropped, so prefer a _bin collation."
+ );
+ }
+
+ kinds.insert(table.clone(), offset_kind_for_data_type(&data_type));
+ }
+ Ok(kinds)
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ /// Best current knowledge of how a table's tracking values compare.
+ async fn tracking_kind(&self, table: &str) -> OffsetKind {
+ self.tracking_kinds
+ .lock()
+ .await
+ .get(table)
+ .copied()
+ .unwrap_or(OffsetKind::Unknown)
+ }
+
+ /// Whether the stored cursor is what limits the rows a poll can see. The built
+ /// polling query always filters on it; a `custom_query` does so only if it
+ /// interpolates `$offset`.
+ fn cursor_gates_query(&self) -> bool {
+ match &self.config.custom_query {
+ Some(query) => query.contains("$offset"),
+ None => true,
+ }
+ }
+
+ /// The offset a poll actually resumes from: the stored cursor, or the configured
+ /// starting point before anything has been stored.
+ fn effective_offset(&self, last_offset: &Option<String>) -> Option<String> {
+ last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let offset_value = self.effective_offset(last_offset).unwrap_or_default();
+ let offset = format_offset_value(&offset_value, kind);
+ let now = Utc::now();
+
+ let q = query
+ .replace("$table", "e_qualified_identifier(table)?)
+ .replace("$offset", &offset)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now_unix", &now.timestamp().to_string())
+ .replace("$now", &now.to_rfc3339());
+
+ Ok(q)
+ }
+
+ /// The query is substituted as text and re-executed once per configured table on
+ /// every poll for the life of the process, so a shape that cannot terminate does
+ /// not fail, it repeats. What can be judged from the text alone is judged here,
+ /// at startup, rather than left to the first poll.
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let tracking_column = self.tracking_column();
+
+ match leading_statement_token(query) {
+ None => {
+ return Err(Error::InitError(
+ "custom_query has no statement: it is empty, or nothing but comments"
+ .to_string(),
+ ));
+ }
+ Some(token) if !READ_STATEMENT_STARTS.contains(&token.as_str()) => {
+ return Err(Error::InitError(format!(
+ "custom_query must be a read, but it starts with '{token}'. It is executed \
+ once per configured table on every poll for the life of the process, so a \
+ statement that writes would re-run forever. Start it with SELECT, WITH or \
+ TABLE."
+ )));
+ }
+ Some(_) => {}
+ }
+
+ // Collapsed so `ORDER\nBY` and `ORDER BY` read the same as `ORDER BY`; the
+ // check hard-fails, so a formatting difference must not decide it.
+ let normalized = query
+ .split_whitespace()
+ .collect::<Vec<_>>()
+ .join(" ")
+ .to_uppercase();
+
+ // Two shapes, two failures. With `$offset` the stored cursor is what limits
+ // the next poll, and it is only the batch's maximum when the rows arrive
+ // ascending, so an unordered query skips rows. Without `$offset` nothing
+ // limits the poll at all: the cursor is written but never read back, so the
+ // same rows return forever unless the rows leave the result set. Only
+ // `delete_after_read` guarantees that on its own - `processed_column` marks
+ // rows the connector never filters on outside the query it builds itself, so
+ // that arm rests on the operator's WHERE and is the weaker of the two.
+ if query.contains("$offset") {
+ if !normalized.contains("ORDER BY") {
+ return Err(Error::InitError(format!(
+ "custom_query filters on $offset but has no ORDER BY. The next offset is the \
+ tracking value of the last row returned, which is only the highest one when \
+ the batch arrives ascending by '{tracking_column}'; without that, rows below \
+ it are skipped permanently. Add `ORDER BY {tracking_column}`."
+ )));
+ }
+ } else if !self.config.delete_after_read.unwrap_or(false)
+ && self.config.processed_column.is_none()
+ {
+ return Err(Error::InitError(format!(
+ "custom_query does not filter on $offset, so every poll returns the same rows and \
+ publishes them again. Interpolate $offset (`WHERE {tracking_column} > $offset`, \
+ which also needs an ORDER BY), or take the rows out of the next poll by setting \
+ delete_after_read, which deletes them, or processed_column, which marks them - \
+ with processed_column the query has to exclude the marked rows itself, since the \
+ connector only appends that filter to the query it builds."
+ )));
+ }
+
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "custom_query uses $table but no tables are configured, so the placeholder \
+ can never resolve; list the tables to poll in `tables`"
+ .to_string(),
+ ));
+ }
+ // Without $table the same SQL is executed once per table, so identical rows
+ // would be emitted repeatedly, each carrying a different table's metadata,
+ // message ID, and tracking offset. Table identity is not recoverable from a
+ // static query, so reject the combination instead of guessing one.
+ if !query.contains("$table") && self.config.tables.len() > 1 {
+ return Err(Error::InitError(format!(
+ "custom_query has no $table placeholder but {} tables are configured ({}); \
+ the same query would run once per table and emit its rows multiple times. \
+ Use $table in the query, or configure a single table.",
+ self.config.tables.len(),
+ self.config.tables.join(", ")
+ )));
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = self.effective_offset(last_offset) {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(&offset, kind)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[PkValue],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({placeholders})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ // The query and its binds are rebuilt on every attempt because with_retry
+ // calls the closure once per try and each `execute` consumes the query.
+ with_retry(
+ || {
+ let mut prepared = sqlx::query(sqlx::AssertSqlSafe(query.as_str()));
+ for id in ids {
+ prepared = match id {
+ PkValue::Text(text) => prepared.bind(text),
+ PkValue::Bytes(bytes) => prepared.bind(bytes),
+ };
+ }
+ prepared.execute(pool)
+ },
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.tracking_column();
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ //
+ // An ordering violation is handled one step further. Retrying it would
+ // re-run the same query, re-decode the same rows, and hit the same failure
+ // forever, so the table is latched off instead of being retried each cycle.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ let poll_number = self.polls.fetch_add(1, Ordering::Relaxed) + 1;
+ let poisoned = { self.poisoned_tables.lock().await.clone() };
+ let mut newly_poisoned: Vec<String> = Vec::new();
+
+ for table in &self.config.tables {
+ if poisoned.contains(table) {
+ continue;
+ }
+
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ if !batch.messages.is_empty() {
+ self.cursor_regressions.lock().await.remove(table);
+ }
+ // A full batch means the LIMIT was hit, so more rows are waiting.
+ any_batch_full |= batch.messages.len() as u32 >= batch_size;
+ batches.push(batch);
+ }
+ Err(FetchError::Ordering(reason) | FetchError::MissingTrackingColumn(reason)) => {
+ let remedy = if self.config.custom_query.is_some() {
+ "with a corrected query"
+ } else {
+ "with a corrected tracking_column, since the connector wrote this query \
+ itself"
+ };
+ error!(
+ "Table '{table}' at offset {}: {reason} No rows were published and the \
+ offset was not advanced. This is deterministic, so the table is disabled \
+ until the connector is restarted {remedy}.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ newly_poisoned.push(table.clone());
+ }
+ Err(FetchError::CursorRegression(reason)) => {
+ // Cleared only by a batch that carried rows, so the run counts polls
+ // since the table last published anything. An idle poll produces as
+ // little as a rejected one does and must not read as recovery.
+ let consecutive = {
+ let mut regressions = self.cursor_regressions.lock().await;
+ let consecutive = regressions.entry(table.clone()).or_insert(0);
+ *consecutive += 1;
+ *consecutive
+ };
+ let offset = last_offset.as_deref().unwrap_or("<start>");
+ if consecutive >= CURSOR_REGRESSION_ERROR_THRESHOLD {
+ error!(
+ "Table '{table}' at offset {offset}: {reason} No rows were published \
+ and the offset was not advanced. This is the {consecutive}th \
+ consecutive poll to end this way, so it is no longer resolving on \
+ its own: the table stays enabled but has published nothing since."
+ );
+ } else {
+ warn!(
+ "Table '{table}' at offset {offset}: {reason} No rows were published \
+ and the offset was not advanced. The table stays enabled, since this \
+ can resolve without a config change."
+ );
+ }
+ }
+ Err(FetchError::StaleOffsetKind(reason)) => {
+ warn!(
+ "Table '{table}' at offset {}: {reason} No rows were published and the \
+ offset was not advanced. The next poll rebuilds the query from the type \
+ the result set reported.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::UnusableTrackingValue(reason)) => {
+ error!(
+ "Table '{table}' at offset {}: {reason} No rows were published and the \
+ offset was not advanced. The table stays enabled so a corrected row is \
+ picked up without a restart, but nothing will be published until then - \
+ this repeats every poll.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::Other(error)) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ if !newly_poisoned.is_empty() {
+ let mut poisoned_tables = self.poisoned_tables.lock().await;
+ poisoned_tables.extend(newly_poisoned);
+ if self
+ .config
+ .tables
+ .iter()
+ .all(|table| poisoned_tables.contains(table))
+ {
+ error!(
+ "All configured tables are disabled because their batches cannot yield a \
+ cursor; {CONNECTOR_NAME} connector with ID: {} is idle until restarted",
+ self.id
+ );
+ }
+ }
+
+ // From the next poll on a disabled table is skipped before anything is logged,
+ // so the one line written when it was disabled can scroll out of the operator's
+ // log window and leave a dark connector looking healthy.
+ if poll_number.is_multiple_of(DISABLED_TABLES_LOG_INTERVAL) {
+ let poisoned_tables = self.poisoned_tables.lock().await;
+ if !poisoned_tables.is_empty() {
+ let mut disabled: Vec<&str> = poisoned_tables.iter().map(String::as_str).collect();
+ disabled.sort_unstable();
+ error!(
+ "{} of {} configured table(s) are still disabled and are not being polled by \
+ {CONNECTOR_NAME} connector with ID: {}, and will stay that way until it is \
+ restarted: {}",
+ disabled.len(),
+ self.config.tables.len(),
+ self.id,
+ disabled.join(", ")
+ );
+ }
+ }
+
+ // Phase 2: commit each table independently. If the mark/delete call fails
+ // for a table we just skip its messages this cycle instead of queuing them
+ // - `mark_or_delete_processed_rows` already retries transient errors, so a
+ // failure here is permanent and the table's rows stay in MySQL to be
+ // retried next cycle, without blocking the others.
+ //
+ // Doesn't cover the case where mark/delete succeeds but the runtime never
+ // gets to publish the batch (crash, send failure, etc) - those rows are
+ // gone from MySQL with nothing delivered.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ let mark_or_delete = self.config.delete_after_read.unwrap_or(false)
+ || self.config.processed_column.is_some();
+
+ for mut batch in batches {
+ if mark_or_delete && !batch.messages.is_empty() && batch.processed_ids.is_empty() {
+ error!(
+ "Table '{}': mark/delete is configured but no primary keys were extracted \
+ from {} row(s), skipping this cycle so rows are not published without being \
+ marked or deleted (check that '{pk_column}' is projected and scalar)",
+ batch.table,
+ batch.messages.len()
+ );
+ continue;
+ }
+
+ if !batch.processed_ids.is_empty()
+ && let Err(error) = self
+ .mark_or_delete_processed_rows(
+ pool,
+ &batch.table,
+ pk_column,
+ &batch.processed_ids,
+ )
+ .await
+ {
+ error!(
+ "Failed to mark or delete processed rows for table '{}', skipping this cycle: {error}",
+ batch.table
+ );
+ continue;
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ state_updates.push((batch.table, offset));
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
+ }
+ }
+
+ self.last_batch_full
+ .store(any_batch_full, Ordering::Relaxed);
+ Ok(messages)
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, FetchError> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query_kind = self.tracking_kind(table).await;
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.substitute_query_params(custom_query, table, last_offset, batch_size, query_kind)?
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size, query_kind)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ // Settled once for the whole batch rather than per row, since the column list
+ // is a property of the result set.
+ if let Some(first) = rows.first() {
+ ensure_tracking_column_projected(
+ first.columns().iter().map(|column| column.name()),
+ tracking_column,
+ )?;
+ }
+
+ let resume_from = if self.cursor_gates_query() {
+ self.effective_offset(last_offset)
+ } else {
+ None
+ };
+
+ // The result set describes what this query actually returned, so it outranks
+ // whatever `information_schema` said at startup and is the only answer
+ // available for a computed column.
+ let kind = match rows.first().and_then(|row| {
+ observed_tracking_kind(row, tracking_column).filter(|observed| *observed != query_kind)
+ }) {
+ Some(observed) => {
+ self.tracking_kinds
+ .lock()
+ .await
+ .insert(table.to_string(), observed);
+ // The query was already sent with the stale kind, so its offset literal
+ // may be quoted for the wrong type.
+ if let Some(offset) = &resume_from
+ && offset_literal_diverged(offset, query_kind, observed)
+ {
+ return Err(FetchError::StaleOffsetKind(format!(
+ "the poll filtered on '{tracking_column}' with {}, written for the \
+ comparison order known before this poll, but the result set reports an \
+ order whose literal is {}. A filter that disagrees with the query's own \
+ ordering strands rows below the cursor permanently, so the batch was \
+ discarded.",
+ format_offset_value(offset, query_kind),
+ format_offset_value(offset, observed)
+ )));
+ }
+ observed
+ }
+ None => query_kind,
+ };
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ let mut ordering = OrderingGuard::new(tracking_column, kind, resume_from);
+ for row in rows {
+ let processed = self.process_row(&row, &table_config).map_err(|e| {
+ error!(
+ "Failed to decode row in table '{table}' (pk {}): {e}",
+ pk_for_log(&row, table_config.pk_column)
+ );
+ e
+ })?;
+
+ let Some(value) = processed.tracking_value else {
+ return Err(unusable_tracking_value(
+ tracking_column,
+ &pk_for_log(&row, table_config.pk_column),
+ ));
+ };
+ ordering
+ .accept(value)
+ .map_err(|violation| match violation {
+ OrderingViolation::WithinBatch(reason) => FetchError::Ordering(reason),
+ OrderingViolation::BelowCursor(reason) => FetchError::CursorRegression(reason),
+ })?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+
+ batch.messages.push(processed.message);
+ }
+ batch.max_offset = ordering.into_cursor();
+
+ Ok(batch)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<PkValue> = None;
+ let mut tracking_value: Option<String> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
+ let mut extracted_payload: Option<Vec<u8>> = None;
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ if name.eq_ignore_ascii_case(config.payload_col) {
+ extracted_payload =
+ Some(self.extract_payload_column(row, i, config.payload_format)?);
+ }
+ if name.eq_ignore_ascii_case(config.tracking_column) {
+ let value = extract_column_value(row, i)?;
+ tracking_value = column_value_as_string(row, i, &value)?;
+ }
+ if name.eq_ignore_ascii_case(config.pk_column) {
+ row_pk = extract_pk_value(row, i)?;
+ }
+ }
+
+ // Every extract_payload_column arm yields Ok for a NULL column, so None
+ // here means the column was absent. Falling back to the whole-row JSON
+ // would publish bytes contradicting the schema derived from payload_format.
+ let payload = extracted_payload.ok_or_else(|| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': payload_column '{}' not present in the result set",
+ config.table, config.payload_col
+ ))
+ })?;
+
+ return Ok(build_processed_row(
+ config.table,
+ payload,
+ tracking_value,
+ row_pk,
+ ));
+ }
+
+ let mut data = serde_json::Map::new();
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ let column_name = if config.snake_case_columns {
+ to_snake_case(name)
+ } else {
+ name.to_string()
+ };
+ let value = extract_column_value(row, i)?;
+ if name.eq_ignore_ascii_case(config.tracking_column) {
+ tracking_value = column_value_as_string(row, i, &value)?;
+ }
+ if name.eq_ignore_ascii_case(config.pk_column) {
+ row_pk = extract_pk_value(row, i)?;
+ }
+ data.insert(column_name, value);
+ }
+
+ let payload = if config.include_metadata {
+ let record = DatabaseRecord {
+ table_name: config.table.to_string(),
+ operation_type: "SELECT".to_string(),
+ timestamp: Utc::now(),
+ data: serde_json::Value::Object(data),
+ old_data: None,
+ };
+ simd_json::to_vec(&record).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': failed to serialize row to JSON: {e}",
+ config.table
+ ))
+ })?
+ } else {
+ simd_json::to_vec(&data).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': failed to serialize row to JSON: {e}",
+ config.table
+ ))
+ })?
+ };
+
+ Ok(build_processed_row(
+ config.table,
+ payload,
+ tracking_value,
+ row_pk,
+ ))
+ }
+}
+
+/// Type-faithful primary-key extraction. Binary columns are taken as raw bytes
+/// rather than routed through the base64 path in `extract_column_value`, so the
+/// value can be bound back into `WHERE pk IN (...)` and match the row. Everything
+/// else takes the string form MySQL parses back to the same value. Returns None
+/// only for a NULL pk, which a real primary key cannot be.
+fn extract_pk_value(row: &MySqlRow, column_index: usize) -> Result<Option<PkValue>, Error> {
+ let column = &row.columns()[column_index];
+ let type_name = column.type_info().name();
+ match type_name {
+ "BINARY" | "VARBINARY" | "TINYBLOB" | "BLOB" | "MEDIUMBLOB" | "LONGBLOB" => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "primary key column '{}' (MySQL type '{type_name}'): {e}",
+ column.name()
+ ))
+ })?;
+ Ok(bytes.map(PkValue::Bytes))
+ }
+ _ => {
+ let value = extract_column_value(row, column_index)?;
+ Ok(column_value_as_string(row, column_index, &value)?.map(PkValue::Text))
+ }
+ }
+}
+
+fn extract_column_value(row: &MySqlRow, column_index: usize) -> Result<serde_json::Value, Error> {
+ let column = &row.columns()[column_index];
+ let type_name = column.type_info().name();
+
+ let to_err = |e: sqlx::Error| {
+ Error::InvalidRecordValue(format!(
+ "column '{}' (MySQL type '{type_name}'): {e}",
+ column.name()
+ ))
+ };
+
+ match type_name {
+ "BOOLEAN" => {
+ let value: Option<bool> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::Bool)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT" => {
+ let value: Option<i8> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT" => {
+ let value: Option<i16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT" | "INT" => {
+ let value: Option<i32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT" => {
+ let value: Option<i64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT UNSIGNED" => {
+ let value: Option<u8> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT UNSIGNED" => {
+ let value: Option<u16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT UNSIGNED" | "INT UNSIGNED" => {
+ let value: Option<u32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT UNSIGNED" => {
+ let value: Option<u64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "FLOAT" => {
+ let value: Option<f32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as f64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DOUBLE" => {
+ let value: Option<f64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DECIMAL" => {
+ let value: Option<rust_decimal::Decimal> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|d| serde_json::Value::String(d.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIT" => {
+ let value: Option<u64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "YEAR" => {
+ let value: Option<u16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATE" => {
+ let value: Option<NaiveDate> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|d| serde_json::Value::String(d.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TIME" => {
+ let value: Option<NaiveTime> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|t| serde_json::Value::String(t.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATETIME" => {
+ let value: Option<NaiveDateTime> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|dt| serde_json::Value::String(dt.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TIMESTAMP" => {
+ let value: Option<DateTime<Utc>> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|dt| serde_json::Value::String(dt.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT" | "MEDIUMTEXT" | "LONGTEXT" | "ENUM" | "SET" => {
+ let value: Option<String> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BINARY" | "VARBINARY" | "TINYBLOB" | "BLOB" | "MEDIUMBLOB" | "LONGBLOB" => {
+ let value: Option<Vec<u8>> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|bytes| {
+ serde_json::Value::String(
+ base64::engine::general_purpose::STANDARD.encode(&bytes),
+ )
+ })
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "JSON" => {
+ let value: Option<serde_json::Value> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value.unwrap_or(serde_json::Value::Null))
+ }
+ "NULL" => Ok(serde_json::Value::Null),
+ _ => {
+ let column_name = column.name();
+ warn!(
+ "Column '{column_name}' has unrecognized MySQL type '{type_name}', \
+ attempting text extraction"
+ );
+ if let Ok(text) = row.try_get::<Option<String>, _>(column_index) {
+ return Ok(text
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null));
+ }
+ if let Ok(bytes) = row.try_get::<Option<Vec<u8>>, _>(column_index) {
+ return Ok(bytes
+ .map(|b| {
+ serde_json::Value::String(
+ base64::engine::general_purpose::STANDARD.encode(&b),
+ )
+ })
+ .unwrap_or(serde_json::Value::Null));
+ }
+ error!(
+ "Column '{column_name}' has unsupported MySQL type '{type_name}', \
+ returning null"
+ );
+ Ok(serde_json::Value::Null)
+ }
+ }
+}
+
+/// Best-effort primary-key value for diagnostics when a row fails to decode.
+/// Falls back to "unknown" if the pk column is absent or itself fails to extract,
+/// so logging a decode failure never masks it with a second error.
+fn pk_for_log(row: &MySqlRow, pk_column: &str) -> String {
+ row.columns()
+ .iter()
+ .position(|column| column.name().eq_ignore_ascii_case(pk_column))
+ .and_then(|index| extract_column_value(row, index).ok())
+ .and_then(|value| value_as_string(&value))
+ .unwrap_or_else(|| "unknown".to_string())
+}
+
+fn value_as_string(value: &serde_json::Value) -> Option<String> {
+ match value {
+ serde_json::Value::String(s) => Some(s.clone()),
+ serde_json::Value::Number(n) => Some(n.to_string()),
+ _ => None,
+ }
+}
+
+/// The string form of a column value that MySQL parses back to the value it came
+/// from, which for `TIMESTAMP` is not its payload form: that one carries chrono's
+/// trailing ` UTC`. A `SELECT` only warns about the superfluous delimiter, but the
+/// same conversion inside the mark/delete DML is an error under the default
+/// `STRICT_TRANS_TABLES`, so both the cursor and the `WHERE pk IN (...)` binding
+/// take this form. Only the payload keeps chrono's rendering.
+fn column_value_as_string(
+ row: &MySqlRow,
+ column_index: usize,
+ value: &serde_json::Value,
+) -> Result<Option<String>, Error> {
+ let column = &row.columns()[column_index];
+ if column.type_info().name() != "TIMESTAMP" {
+ return Ok(value_as_string(value));
+ }
+
+ let timestamp: Option<DateTime<Utc>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "tracking column '{}' (MySQL type 'TIMESTAMP'): {e}",
+ column.name()
+ ))
+ })?;
+ Ok(timestamp.map(timestamp_cursor))
+}
+
+/// Fixed width, so the collation comparison a temporal cursor is ordered by agrees
+/// with the instants the values stand for.
+fn timestamp_cursor(timestamp: DateTime<Utc>) -> String {
+ timestamp.format("%Y-%m-%d %H:%M:%S%.6f").to_string()
+}
+
+fn build_processed_row(
+ table: &str,
+ payload: Vec<u8>,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+) -> ProcessedRow {
+ let now = Utc::now().timestamp_micros() as u64;
+ // The key is the pk's string form, so changing how a column type renders re-keys
+ // every row of that type. Once connectors are in the field that breaks dedup for
+ // any row replayed across the change, so weigh it before touching
+ // `column_value_as_string`.
+ let pk_key = row_pk.as_ref().map(PkValue::as_key);
+ let id = message_id(table, pk_key.as_deref().or(tracking_value.as_deref()));
+ ProcessedRow {
+ message: ProducedMessage {
+ id: Some(id),
+ headers: None,
+ checksum: None,
+ timestamp: Some(now),
+ origin_timestamp: Some(now),
+ payload,
+ },
+ tracking_value,
+ row_pk,
+ }
+}
+
+/// Deterministic message id so a row replayed after a restart keeps the same id
+/// and downstream can dedup it. Derived from the table plus the row's stable key
+/// (its primary key, or the unique/monotonic tracking value). The random branch
+/// is unreachable for a contract-compliant tracking column - it only avoids a
+/// panic if a row yields no usable key, at the cost of that one row not being
+/// idempotent on replay.
+fn message_id(table: &str, key: Option<&str>) -> u128 {
+ match key {
+ Some(key) => {
+ let name = format!("{table}\0{key}");
+ Uuid::new_v5(&MESSAGE_ID_NAMESPACE, name.as_bytes()).as_u128()
+ }
+ None => {
+ warn!(
+ "Row in table '{table}' has no primary-key or tracking value; \
+ using a random message id (replays will not be idempotent)"
+ );
+ Uuid::new_v4().as_u128()
+ }
+ }
+}
+
+fn to_snake_case(input: &str) -> String {
+ let mut result = String::new();
+ let mut prev_was_uppercase = false;
+ for (i, ch) in input.chars().enumerate() {
+ if ch.is_uppercase() {
+ if i > 0 && !prev_was_uppercase {
+ result.push('_');
+ }
+ if let Some(lc) = ch.to_lowercase().next() {
+ result.push(lc);
+ } else {
+ result.push(ch);
+ }
+ prev_was_uppercase = true;
+ } else {
+ result.push(ch);
+ prev_was_uppercase = false;
+ }
+ }
+ result
+}
+
+fn redact_connection_string(conn_str: &str) -> String {
+ if let Some(scheme_end) = conn_str.find("://") {
+ let scheme = &conn_str[..scheme_end + 3];
+ let rest = &conn_str[scheme_end + 3..];
+ let preview: String = rest.chars().take(3).collect();
+ return format!("{scheme}{preview}***");
+ }
+ let preview: String = conn_str.chars().take(3).collect();
+ format!("{preview}***")
+}
+
+fn quote_identifier(name: &str) -> Result<String, Error> {
Review Comment:
quote_identifier/quote_qualified_identifier (lib.rs:1904/1919) are injection-safe. distsys traced them end-to-end and attempted a concrete breakout: both reject embedded NUL (the one character MySQL genuinely forbids in a quoted identifier), double every embedded backtick correctly, and always wrap in backticks regardless of ANSI_QUOTES mode. A crafted input like a`.`b was traced through to confirm it still parses as quoted identifiers, not a breakout. storage independently reached the same conclusion via its own trace, with one residual edge case: under NO_BACKSLASH_ESCAPES SQL mode, the literal-value escaping in format_offset_value (not the identifier quoting) can corrupt a cursor's backslash content, causing a wrong comparison — a correctness bug, not an injection vector, since it still produces a validly-escaped literal under that mode. The actual injection risk in this PR is not the identifier quoting — it's the substitution-order bug above. Worth being precise about this
distinction since it's easy to conflate "has SQL built via format!" with "is injectable" — the quoting discipline here is sound; the token-substitution ordering is not
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on PR #3568:
URL: https://github.com/apache/iggy/pull/3568#issuecomment-6253641235
/ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r4303313533
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,4071 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source,
+ source::SourceBatchResult, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+/// Upper bound on applying a batch's staged mark/delete after its Ack. The SDK waits
+/// `BATCH_RESULT_TIMEOUT` (30s) for the result, so this has to finish well inside it.
+const ACK_BATCH_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// MySQL rejects a prepared statement with more placeholders than this (error 1390), and
+/// the mark/delete binds one per row of the batch.
+const MAX_CLEANUP_BATCH_SIZE: u32 = 65_535;
+
+/// How many mark/delete statements in a row may fail before the connector stops. The
+/// operations stay in the persisted state and run again on restart.
+const MAX_CONSECUTIVE_CLEANUP_FAILURES: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ pending_batch: Mutex<Option<PendingBatch>>,
+ consecutive_cleanup_failures: AtomicU32,
+ verified_cleanup_keys: Mutex<HashSet<String>>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some("it has too few distinct values to be unique per row");
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some("binary values are compared as base64 text, which does not keep byte order");
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some("MySQL orders these by declaration index, not by label");
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL \
+ or not an ordered scalar (tinyint(1), JSON)"
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "row '{value}' is below the stored offset '{previous}' for tracking_column \
+ '{column}'. Filter the query with `{column} > $offset`"
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. Order the query ascending by '{column}'"
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+ /// Mark/delete work for a batch the runtime has already sent. It is persisted with
+ /// the batch's offsets so a crash between the send and the cleanup reruns it.
+ #[serde(default)]
+ pending_operations: Vec<PendingOperation>,
+}
+
+/// The rows of one table to mark or delete once their batch is acknowledged.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct PendingOperation {
+ table: String,
+ ids: Vec<PkValue>,
+ target: CleanupTarget,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct CleanupTarget {
+ key_column: String,
+ action: CleanupAction,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum CleanupAction {
+ Delete,
+ MarkProcessed { column: String },
+}
+
+/// The candidate state of the batch most recently handed to the runtime. Live state
+/// only takes it over on Ack, so a Nack leaves offsets and rows where they were.
+#[derive(Debug, Clone)]
+struct PendingBatch {
+ state: State,
+ /// The runtime has delivered the batch but its cleanup has not finished yet.
+ acknowledged: bool,
+ /// Cleanup finished and the next poll must persist `state` without the operations,
+ /// so a restart does not run them again.
+ retiring: bool,
+}
+
+struct PolledBatch {
+ messages: Vec<ProducedMessage>,
+ pending: Option<PendingBatch>,
+}
+
+enum CleanupKeyCheck {
+ Unique,
+ NotUnique,
+ TableMissing,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty, list the table even when custom_query hardcodes it"
+ .to_string(),
+ ));
+ }
+
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ if let Some(batch_size) = self.config.batch_size
+ && batch_size > MAX_CLEANUP_BATCH_SIZE
+ && self.cleanup_target().is_some()
+ {
+ return Err(Error::InitError(format!(
+ "batch_size {batch_size} exceeds {MAX_CLEANUP_BATCH_SIZE} with delete_after_read or \
+ processed_column set, MySQL allows at most {MAX_CLEANUP_BATCH_SIZE} placeholders \
+ in the statement that marks or deletes a batch"
+ )));
+ }
+
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "name the column custom_query orders by and filters with $offset"
+ } else {
+ "use a unique, monotonically increasing column"
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set, {guidance}"
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.validate_pending_cleanup()?;
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+ self.validate_cleanup_keys().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ if let Some(state) = self.retiring_state().await {
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: Some(self.serialize_pending_state(&state)?),
+ });
+ }
+
+ // Skip the pacing delay while draining a backlog, but never while a cleanup
+ // retry is pending, or a failing mark/delete would spin without a pause.
+ let cleanup_pending = self.pending_batch.lock().await.is_some();
+ if cleanup_pending || !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ {
+ let pending = self.pending_batch.lock().await;
+ if let Some(pending) = pending.as_ref() {
+ if !pending.acknowledged {
+ error!(
+ "{CONNECTOR_NAME} connector ID: {} was polled while a batch was still in flight",
+ self.id
+ );
+ return Err(Error::InvalidState);
+ }
+ // An empty batch whose Ack retries the cleanup the last one left behind.
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: None,
+ });
+ }
+ }
+
+ let polled = self.poll_tables().await?;
+
+ let processed_rows = match &polled.pending {
+ Some(pending) => pending.state.processed_rows,
+ None => self.state.lock().await.processed_rows,
+ };
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ }
+
+ // Idle cycles stage nothing, so return None to let the runtime skip the fsync.
+ let persisted_state = polled
+ .pending
+ .as_ref()
+ .map(|pending| self.serialize_pending_state(&pending.state))
+ .transpose()?;
+ *self.pending_batch.lock().await = polled.pending;
+
+ Ok(ProducedMessages {
+ schema,
+ messages: polled.messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> {
+ if result == SourceBatchResult::Nack {
+ // Dropping the candidate leaves the live offsets where they were, so the
+ // next poll reads the same rows again. Delivered batches keep their cleanup.
+ let mut pending = self.pending_batch.lock().await;
+ if pending
+ .as_ref()
+ .is_some_and(|pending| !pending.acknowledged && !pending.retiring)
+ {
+ pending.take();
+ }
+ return Ok(());
+ }
+
+ let Some(PendingBatch { mut state, .. }) = self.pending_batch.lock().await.clone() else {
+ return Ok(());
+ };
+
+ let operations = std::mem::take(&mut state.pending_operations);
+ let had_operations = !operations.is_empty();
+ let deadline = tokio::time::Instant::now() + ACK_BATCH_TIMEOUT;
+ let failed_operations = self.apply_pending_operations(operations, deadline).await?;
+
+ if failed_operations.is_empty() {
+ *self.state.lock().await = state.clone();
+ let mut pending = self.pending_batch.lock().await;
+ if had_operations {
+ *pending = Some(PendingBatch {
+ state,
+ acknowledged: false,
+ retiring: true,
+ });
+ } else {
+ pending.take();
+ }
+ } else {
+ state.pending_operations = failed_operations;
+ *self.pending_batch.lock().await = Some(PendingBatch {
+ state,
+ acknowledged: true,
+ retiring: false,
+ });
+ }
+
+ Ok(())
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ let state = restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ pending_operations: Vec::new(),
+ });
+ // The runtime only persists a batch's operations after sending it, so restored
+ // operations belong to a delivered batch and only the cleanup is left to do.
+ let pending_batch = (!state.pending_operations.is_empty()).then(|| PendingBatch {
+ state: state.clone(),
+ acknowledged: true,
+ retiring: false,
+ });
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(state),
+ pending_batch: Mutex::new(pending_batch),
+ consecutive_cleanup_failures: AtomicU32::new(0),
+ verified_cleanup_keys: Mutex::new(HashSet::new()),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ poisoned_tables: Mutex::new(HashSet::new()),
+ tracking_kinds: Mutex::new(HashMap::new()),
+ polls: AtomicU64::new(0),
+ cursor_regressions: Mutex::new(HashMap::new()),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ // `format_offset_value` escapes backslashes, which under NO_BACKSLASH_ESCAPES
+ // would reach MySQL as literal characters and filter on the wrong cursor.
+ .after_connect(|connection, _| {
+ Box::pin(async move {
+ sqlx::query(
+ "SET SESSION sql_mode = REPLACE(@@SESSION.sql_mode, 'NO_BACKSLASH_ESCAPES', '')",
+ )
+ .execute(&mut *connection)
+ .await?;
+ Ok(())
+ })
+ })
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn tracking_column(&self) -> &str {
+ self.config.tracking_column.as_deref().unwrap_or("id")
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn resolve_tracking_columns(&self) -> Result<HashMap<String, OffsetKind>, Error> {
+ // Whether the connector owns the query, and can therefore hold the schema to
+ // what that query will do with it.
+ let strict = self.config.custom_query.is_none();
+ let pool = self.get_pool()?;
+ let tracking_column = self.tracking_column();
+ let mut kinds = HashMap::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ }
+
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ }
+
+ warn!(
+ "tracking_column '{tracking_column}' is not a column of table '{table}'; \
+ assuming custom_query projects it. Its comparison order is resolved from \
+ the first result set instead."
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ // The column's type survives a plain projection, so it is worth rejecting
+ // whatever the query looks like. A custom_query aliasing an expression onto
+ // a name the table also uses is the one false positive, hence the hint.
+ if let Some(reason) = unusable_tracking_type(&data_type) {
+ let aliasing_hint = if self.config.custom_query.is_some() {
+ " If custom_query aliases a different expression to this name, rename the \
+ alias so it does not collide with the column."
+ } else {
+ ""
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': {reason}. Use an integer, timestamp, or string \
+ column.{aliasing_hint}"
+ )));
+ }
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; a row \
+ whose value is NULL cannot produce a cursor and will fail that table's batch. \
+ Have custom_query exclude those rows, or declare the column NOT NULL."
+ );
+ }
+
+ if is_collation_ordered(&data_type) {
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': values must be unique and increase in its collation order \
+ (zero-pad numbers, prefer a _bin collation)"
+ );
+ }
+
+ kinds.insert(table.clone(), offset_kind_for_data_type(&data_type));
+ }
+ Ok(kinds)
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn serialize_pending_state(&self, state: &State) -> Result<ConnectorState, Error> {
+ self.serialize_state(state).ok_or_else(|| {
+ Error::Serialization(format!("failed to serialize {CONNECTOR_NAME} state"))
+ })
+ }
+
+ async fn retiring_state(&self) -> Option<State> {
+ self.pending_batch
+ .lock()
+ .await
+ .as_ref()
+ .filter(|pending| pending.retiring)
+ .map(|pending| pending.state.clone())
+ }
+
+ fn primary_key_column(&self) -> &str {
+ self.config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(self.tracking_column())
+ }
+
+ fn cleanup_target(&self) -> Option<CleanupTarget> {
+ let action = if self.config.delete_after_read.unwrap_or(false) {
+ CleanupAction::Delete
+ } else if let Some(column) = &self.config.processed_column {
+ CleanupAction::MarkProcessed {
+ column: column.clone(),
+ }
+ } else {
+ return None;
+ };
+ Some(CleanupTarget {
+ key_column: self.primary_key_column().to_string(),
+ action,
+ })
+ }
+
+ /// Restored operations were staged under the config of the run that wrote them.
+ /// Running them under a different key or action would touch rows nobody read.
+ fn validate_pending_cleanup(&mut self) -> Result<(), Error> {
+ let configured = self.cleanup_target();
+ for operation in &self.state.get_mut().pending_operations {
+ if configured.as_ref() != Some(&operation.target) {
+ return Err(Error::InitError(format!(
+ "the cleanup configuration changed while cleanup of {} delivered row(s) in \
+ table '{}' is pending; restore primary_key_column '{}' and action {:?}",
+ operation.ids.len(),
+ operation.table,
+ operation.target.key_column,
+ operation.target.action
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ /// `WHERE key IN (...)` on a key that repeats marks or deletes every row sharing a
+ /// value, including ones that were never read. Tables that don't exist yet are
+ /// checked by the first poll that stages cleanup for them.
+ async fn validate_cleanup_keys(&mut self) -> Result<(), Error> {
+ let Some(target) = self.cleanup_target() else {
+ return Ok(());
+ };
+ let pool = self.get_pool()?;
+ let mut verified = HashSet::with_capacity(self.config.tables.len());
+ for table in &self.config.tables {
+ let check = cleanup_key_check(pool, table, &target.key_column)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to check primary_key_column '{}' on table '{table}': {e}",
+ target.key_column
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ verified.insert(table.clone());
+ }
+ CleanupKeyCheck::NotUnique => {
+ return Err(Error::InitError(not_unique_key_message(
+ table,
+ &target.key_column,
+ )));
+ }
+ CleanupKeyCheck::TableMissing => {}
+ }
+ }
+ *self.verified_cleanup_keys.get_mut() = verified;
+ Ok(())
+ }
+
+ async fn cleanup_key_is_unique(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ key_column: &str,
+ ) -> Result<bool, Error> {
+ if self.verified_cleanup_keys.lock().await.contains(table) {
+ return Ok(true);
+ }
+ let check = cleanup_key_check(pool, table, key_column)
+ .await
+ .map_err(|e| {
+ Error::Connection(format!(
+ "failed to check primary_key_column '{key_column}' on table '{table}': {e}"
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ self.verified_cleanup_keys
+ .lock()
+ .await
+ .insert(table.to_string());
+ Ok(true)
+ }
+ CleanupKeyCheck::NotUnique => Ok(false),
+ CleanupKeyCheck::TableMissing => Err(Error::Connection(format!(
+ "table '{table}' disappeared before its primary_key_column could be checked"
+ ))),
+ }
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ /// Best current knowledge of how a table's tracking values compare.
+ async fn tracking_kind(&self, table: &str) -> OffsetKind {
+ self.tracking_kinds
+ .lock()
+ .await
+ .get(table)
+ .copied()
+ .unwrap_or(OffsetKind::Unknown)
+ }
+
+ /// Whether the stored cursor is what limits the rows a poll can see. The built
+ /// polling query always filters on it; a `custom_query` does so only if it
+ /// interpolates `$offset`.
+ fn cursor_gates_query(&self) -> bool {
+ match &self.config.custom_query {
+ Some(query) => query.contains("$offset"),
+ None => true,
+ }
+ }
+
+ /// The offset a poll actually resumes from: the stored cursor, or the configured
+ /// starting point before anything has been stored.
+ fn effective_offset(&self, last_offset: &Option<String>) -> Option<String> {
+ last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let offset_value = self.effective_offset(last_offset).unwrap_or_default();
+ let offset = format_offset_value(&offset_value, kind);
+ let quoted_table = quote_qualified_identifier(table)?;
+ let limit = batch_size.to_string();
+ let now = Utc::now();
+ let now_unix = now.timestamp().to_string();
+ let now = now.to_rfc3339();
+ let replacements = [
+ ("$table", quoted_table.as_str()),
+ ("$offset", offset.as_str()),
+ ("$limit", limit.as_str()),
+ ("$now_unix", now_unix.as_str()),
+ ("$now", now.as_str()),
+ ];
+
+ Ok(substitute_query_tokens(query, &replacements))
+ }
+
+ /// The query is substituted as text and re-executed once per configured table on
+ /// every poll for the life of the process, so a shape that cannot terminate does
+ /// not fail, it repeats. What can be judged from the text alone is judged here,
+ /// at startup, rather than left to the first poll.
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let tracking_column = self.tracking_column();
+
+ match leading_statement_token(query) {
+ None => {
+ return Err(Error::InitError(
+ "custom_query has no statement: it is empty, or nothing but comments"
+ .to_string(),
+ ));
+ }
+ Some(token) if !READ_STATEMENT_STARTS.contains(&token.as_str()) => {
+ return Err(Error::InitError(format!(
+ "custom_query must be a read starting with SELECT, WITH or TABLE, found \
+ '{token}'"
+ )));
+ }
+ Some(_) => {}
+ }
+
+ // Collapsed so `ORDER\nBY` and `ORDER BY` read the same as `ORDER BY`; the
+ // check hard-fails, so a formatting difference must not decide it.
+ let normalized = query
+ .split_whitespace()
+ .collect::<Vec<_>>()
+ .join(" ")
+ .to_uppercase();
+
+ if query.contains("$offset") {
+ if !normalized.contains("ORDER BY") {
+ return Err(Error::InitError(format!(
+ "custom_query filters on $offset but has no ORDER BY, add \
+ `ORDER BY {tracking_column}`"
+ )));
+ }
+ } else if !self.config.delete_after_read.unwrap_or(false)
+ && self.config.processed_column.is_none()
+ {
+ return Err(Error::InitError(format!(
+ "custom_query must filter with $offset (`WHERE {tracking_column} > $offset`), or \
+ set delete_after_read or processed_column"
+ )));
+ }
+
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "custom_query uses $table but no tables are configured".to_string(),
+ ));
+ }
+ // Without $table the same SQL is executed once per table, so identical rows
+ // would be emitted repeatedly, each carrying a different table's metadata,
+ // message ID, and tracking offset. Table identity is not recoverable from a
+ // static query, so reject the combination instead of guessing one.
+ if !query.contains("$table") && self.config.tables.len() > 1 {
+ return Err(Error::InitError(format!(
+ "custom_query has no $table placeholder but {} tables are configured ({}), use \
+ $table in the query or configure a single table",
+ self.config.tables.len(),
+ self.config.tables.join(", ")
+ )));
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = self.effective_offset(last_offset) {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(&offset, kind)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn apply_pending_operations(
+ &self,
+ operations: Vec<PendingOperation>,
+ deadline: tokio::time::Instant,
+ ) -> Result<Vec<PendingOperation>, Error> {
+ let mut failed_operations = Vec::new();
+ let mut operations = operations.into_iter();
+ while let Some(operation) = operations.next() {
+ if tokio::time::Instant::now() >= deadline {
+ failed_operations.push(operation);
+ failed_operations.extend(operations);
+ break;
+ }
+ let result = tokio::time::timeout_at(deadline, async {
+ let pool = self.get_pool()?;
+ self.mark_or_delete_processed_rows(pool, &operation).await
+ })
+ .await;
+ match result {
+ Ok(Ok(())) => {
+ self.consecutive_cleanup_failures
+ .store(0, Ordering::Relaxed);
+ }
+ Ok(Err(error)) => {
+ self.record_cleanup_failure(&operation, &error)?;
+ failed_operations.push(operation);
+ }
+ Err(_) => {
+ self.record_cleanup_failure(
+ &operation,
+ "cleanup exceeded the 10s budget for one Ack",
+ )?;
+ failed_operations.push(operation);
+ }
+ }
+ }
+ Ok(failed_operations)
+ }
+
+ fn record_cleanup_failure(
+ &self,
+ operation: &PendingOperation,
+ error: impl std::fmt::Display,
+ ) -> Result<(), Error> {
+ let consecutive_failures = self
+ .consecutive_cleanup_failures
+ .fetch_add(1, Ordering::Relaxed)
+ .saturating_add(1);
+ error!(
+ "Failed to mark or delete {} delivered row(s) in table '{}' for {CONNECTOR_NAME} \
+ connector ID: {}, retrying on the next poll ({consecutive_failures} consecutive \
+ failures): {error}",
+ operation.ids.len(),
+ operation.table,
+ self.id
+ );
+ if consecutive_failures >= MAX_CONSECUTIVE_CLEANUP_FAILURES {
+ return Err(Error::Connection(format!(
+ "stopping {CONNECTOR_NAME} connector ID {} after {consecutive_failures} \
+ consecutive mark/delete failures; the cleanup stays in the saved state and \
+ runs again on restart",
+ self.id
+ )));
+ }
+ Ok(())
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ operation: &PendingOperation,
+ ) -> Result<(), Error> {
+ let PendingOperation { table, ids, target } = operation;
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(&target.key_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = match &target.action {
+ CleanupAction::Delete => {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
+ }
+ CleanupAction::MarkProcessed { column } => {
+ let quoted_processed = quote_identifier(column)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({placeholders})"
+ )
+ }
+ };
+
+ // The query and its binds are rebuilt on every attempt because with_retry
+ // calls the closure once per try and each `execute` consumes the query.
+ with_retry(
+ || {
+ let mut prepared = sqlx::query(sqlx::AssertSqlSafe(query.as_str()));
+ for id in ids {
+ prepared = match id {
+ PkValue::Text(text) => prepared.bind(text),
+ PkValue::Bytes(bytes) => prepared.bind(bytes),
+ };
+ }
+ prepared.execute(pool)
+ },
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<PolledBatch, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.tracking_column();
+ let pk_column = self.primary_key_column();
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ //
+ // An ordering violation is handled one step further. Retrying it would
+ // re-run the same query, re-decode the same rows, and hit the same failure
+ // forever, so the table is latched off instead of being retried each cycle.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ let poll_number = self.polls.fetch_add(1, Ordering::Relaxed) + 1;
+ let poisoned = { self.poisoned_tables.lock().await.clone() };
+ let mut newly_poisoned: Vec<String> = Vec::new();
+
+ for table in &self.config.tables {
+ if poisoned.contains(table) {
+ continue;
+ }
+
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ if !batch.messages.is_empty() {
+ self.cursor_regressions.lock().await.remove(table);
+ }
+ // A full batch means the LIMIT was hit, so more rows are waiting.
+ any_batch_full |= batch.messages.len() as u32 >= batch_size;
+ batches.push(batch);
+ }
+ Err(FetchError::Ordering(reason) | FetchError::MissingTrackingColumn(reason)) => {
+ error!(
+ "Table '{table}' at offset {}: {reason}. Table disabled until restart.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ newly_poisoned.push(table.clone());
+ }
+ Err(FetchError::CursorRegression(reason)) => {
+ // Cleared only by a batch that carried rows, so the run counts polls
+ // since the table last published anything. An idle poll produces as
+ // little as a rejected one does and must not read as recovery.
+ let consecutive = {
+ let mut regressions = self.cursor_regressions.lock().await;
+ let consecutive = regressions.entry(table.clone()).or_insert(0);
+ *consecutive += 1;
+ *consecutive
+ };
+ let offset = last_offset.as_deref().unwrap_or("<start>");
+ if consecutive >= CURSOR_REGRESSION_ERROR_THRESHOLD {
+ error!(
+ "Table '{table}' at offset {offset}: {reason}. Skipping this poll \
+ ({consecutive} consecutive polls)."
+ );
+ } else {
+ warn!("Table '{table}' at offset {offset}: {reason}. Skipping this poll.");
+ }
+ }
+ Err(FetchError::StaleOffsetKind(reason)) => {
+ warn!(
+ "Table '{table}' at offset {}: {reason}. Retrying on the next poll.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::UnusableTrackingValue(reason)) => {
+ error!(
+ "Table '{table}' at offset {}: {reason}. Retrying each poll until the row \
+ is fixed.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::Other(error)) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ if !newly_poisoned.is_empty() {
+ let mut poisoned_tables = self.poisoned_tables.lock().await;
+ poisoned_tables.extend(newly_poisoned);
+ if self
+ .config
+ .tables
+ .iter()
+ .all(|table| poisoned_tables.contains(table))
+ {
+ error!(
+ "All configured tables are disabled because their batches cannot yield a \
+ cursor; {CONNECTOR_NAME} connector with ID: {} is idle until restarted",
+ self.id
+ );
+ }
+ }
+
+ // From the next poll on a disabled table is skipped before anything is logged,
+ // so the one line written when it was disabled can scroll out of the operator's
+ // log window and leave a dark connector looking healthy.
+ if poll_number.is_multiple_of(DISABLED_TABLES_LOG_INTERVAL) {
+ let poisoned_tables = self.poisoned_tables.lock().await;
+ if !poisoned_tables.is_empty() {
+ let mut disabled: Vec<&str> = poisoned_tables.iter().map(String::as_str).collect();
+ disabled.sort_unstable();
+ error!(
+ "{} of {} configured table(s) are still disabled and are not being polled by \
+ {CONNECTOR_NAME} connector with ID: {}, and will stay that way until it is \
+ restarted: {}",
+ disabled.len(),
+ self.config.tables.len(),
+ self.id,
+ disabled.join(", ")
+ );
+ }
+ }
+
+ // Phase 2: stage each table's batch. Nothing touches MySQL or the live state
+ // here: the offsets and the mark/delete are applied in `on_batch_result` once
+ // the runtime has sent the batch and saved this candidate state, and a Nack
+ // throws the candidate away so the same rows are read again.
+ let cleanup_target = self.cleanup_target();
+ let mut candidate_state = self.state.lock().await.clone();
+ let mut operations = Vec::new();
+ let mut messages = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if let Some(target) = &cleanup_target
+ && !batch.messages.is_empty()
+ {
+ if batch.processed_ids.is_empty() {
+ error!(
+ "Table '{}': mark/delete is configured but no primary keys were extracted \
+ from {} row(s), skipping this cycle so rows are not published without \
+ being marked or deleted (check that '{pk_column}' is projected and scalar)",
+ batch.table,
+ batch.messages.len()
+ );
+ continue;
+ }
+
+ match self
+ .cleanup_key_is_unique(pool, &batch.table, &target.key_column)
+ .await
+ {
+ Ok(true) => {}
+ Ok(false) => {
+ error!(
+ "{}. Table disabled until restart.",
+ not_unique_key_message(&batch.table, &target.key_column)
+ );
+ self.poisoned_tables.lock().await.insert(batch.table);
+ continue;
+ }
+ Err(error) => {
+ error!("Table '{}' skipped this cycle: {error}", batch.table);
+ continue;
+ }
+ }
+
+ operations.push(PendingOperation {
+ table: batch.table.clone(),
+ ids: std::mem::take(&mut batch.processed_ids),
+ target: target.clone(),
+ });
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ candidate_state.tracking_offsets.insert(batch.table, offset);
+ }
+ }
+
+ let pending = (total_processed > 0).then(|| {
+ candidate_state.processed_rows += total_processed;
+ candidate_state.pending_operations = operations;
+ PendingBatch {
+ state: candidate_state,
+ acknowledged: false,
+ retiring: false,
+ }
+ });
+
+ self.last_batch_full
+ .store(any_batch_full, Ordering::Relaxed);
+ Ok(PolledBatch { messages, pending })
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, FetchError> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query_kind = self.tracking_kind(table).await;
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.substitute_query_params(custom_query, table, last_offset, batch_size, query_kind)?
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size, query_kind)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ // Settled once for the whole batch rather than per row, since the column list
+ // is a property of the result set.
+ if let Some(first) = rows.first() {
+ ensure_tracking_column_projected(
+ first.columns().iter().map(|column| column.name()),
+ tracking_column,
+ )?;
+ }
+
+ let resume_from = if self.cursor_gates_query() {
+ self.effective_offset(last_offset)
+ } else {
+ None
+ };
+
+ // The result set describes what this query actually returned, so it outranks
+ // whatever `information_schema` said at startup and is the only answer
+ // available for a computed column.
+ let kind = match rows.first().and_then(|row| {
+ observed_tracking_kind(row, tracking_column).filter(|observed| *observed != query_kind)
+ }) {
+ Some(observed) => {
+ self.tracking_kinds
+ .lock()
+ .await
+ .insert(table.to_string(), observed);
+ // The query was already sent with the stale kind, so its offset literal
+ // may be quoted for the wrong type.
+ if let Some(offset) = &resume_from
+ && offset_literal_diverged(offset, query_kind, observed)
+ {
+ return Err(FetchError::StaleOffsetKind(format!(
+ "tracking_column '{tracking_column}' type differs from the one detected \
+ at startup (filtered with {}, expected {}), batch discarded",
+ format_offset_value(offset, query_kind),
+ format_offset_value(offset, observed)
+ )));
+ }
+ observed
+ }
+ None => query_kind,
+ };
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ // Every row of a result set shares its columns, so the published key names are
+ // derived once per batch instead of once per row.
+ let column_names: Vec<String> = rows
+ .first()
+ .map(|row| {
+ row.columns()
+ .iter()
+ .map(|column| {
+ if table_config.snake_case_columns {
+ to_snake_case(column.name())
+ } else {
+ column.name().to_string()
+ }
+ })
+ .collect()
+ })
+ .unwrap_or_default();
+ let mut ordering = OrderingGuard::new(tracking_column, kind, resume_from);
+ for row in rows {
+ let processed = self
+ .process_row(&row, &table_config, &column_names)
+ .map_err(|e| {
+ error!(
+ "Failed to decode row in table '{table}' (pk {}): {e}",
+ pk_for_log(&row, table_config.pk_column)
+ );
+ e
+ })?;
+
+ let Some(value) = processed.tracking_value else {
+ return Err(unusable_tracking_value(
+ tracking_column,
+ &pk_for_log(&row, table_config.pk_column),
+ ));
+ };
+ ordering
+ .accept(value)
+ .map_err(|violation| match violation {
+ OrderingViolation::WithinBatch(reason) => FetchError::Ordering(reason),
+ OrderingViolation::BelowCursor(reason) => FetchError::CursorRegression(reason),
+ })?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+
+ batch.messages.push(processed.message);
+ }
+ batch.max_offset = ordering.into_cursor();
+
+ Ok(batch)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ column_names: &[String],
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<PkValue> = None;
+ let mut tracking_value: Option<String> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
+ let mut extracted_payload: Option<Vec<u8>> = None;
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ if name.eq_ignore_ascii_case(config.payload_col) {
+ extracted_payload =
+ Some(self.extract_payload_column(row, i, config.payload_format)?);
+ }
+ let is_tracking = name.eq_ignore_ascii_case(config.tracking_column);
+ let is_pk = name.eq_ignore_ascii_case(config.pk_column);
+ if is_tracking {
+ let value = extract_column_value(row, i)?;
+ tracking_value = column_value_as_string(row, i, &value)?;
+ if is_pk {
+ row_pk = pk_value_from_decoded(row, i, &value)?;
+ }
+ } else if is_pk {
+ row_pk = extract_pk_value(row, i)?;
+ }
+ }
+
+ // Every extract_payload_column arm yields Ok for a NULL column, so None
+ // here means the column was absent. Falling back to the whole-row JSON
+ // would publish bytes contradicting the schema derived from payload_format.
+ let payload = extracted_payload.ok_or_else(|| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': payload_column '{}' not present in the result set",
+ config.table, config.payload_col
+ ))
+ })?;
+
+ return Ok(build_processed_row(
+ config.table,
+ payload,
+ tracking_value,
+ row_pk,
+ ));
+ }
+
+ let mut data = serde_json::Map::new();
+ for ((i, column), column_name) in row.columns().iter().enumerate().zip(column_names) {
+ let name = column.name();
+ let value = extract_column_value(row, i)?;
+ if name.eq_ignore_ascii_case(config.tracking_column) {
+ tracking_value = column_value_as_string(row, i, &value)?;
+ }
+ if name.eq_ignore_ascii_case(config.pk_column) {
+ row_pk = pk_value_from_decoded(row, i, &value)?;
+ }
+ data.insert(column_name.clone(), value);
+ }
+
+ let payload = if config.include_metadata {
+ let record = DatabaseRecord {
+ table_name: config.table.to_string(),
+ operation_type: "SELECT".to_string(),
+ timestamp: Utc::now(),
+ data: serde_json::Value::Object(data),
+ old_data: None,
+ };
+ simd_json::to_vec(&record).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': failed to serialize row to JSON: {e}",
+ config.table
+ ))
+ })?
+ } else {
+ simd_json::to_vec(&data).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': failed to serialize row to JSON: {e}",
+ config.table
+ ))
+ })?
+ };
+
+ Ok(build_processed_row(
+ config.table,
+ payload,
+ tracking_value,
+ row_pk,
+ ))
+ }
+}
+
+/// Type-faithful primary-key extraction. Binary columns are taken as raw bytes
+/// rather than routed through the base64 path in `extract_column_value`, so the
+/// value can be bound back into `WHERE pk IN (...)` and match the row. Everything
+/// else takes the string form MySQL parses back to the same value. Returns None
+/// only for a NULL pk, which a real primary key cannot be.
+fn extract_pk_value(row: &MySqlRow, column_index: usize) -> Result<Option<PkValue>, Error> {
+ if is_binary_column(row, column_index) {
+ return binary_pk_value(row, column_index);
+ }
+ let value = extract_column_value(row, column_index)?;
+ pk_value_from_decoded(row, column_index, &value)
+}
+
+/// Like `extract_pk_value` for a column the caller has already decoded, so the
+/// common key column is not decoded a second time per row.
+fn pk_value_from_decoded(
+ row: &MySqlRow,
+ column_index: usize,
+ value: &serde_json::Value,
+) -> Result<Option<PkValue>, Error> {
+ if is_binary_column(row, column_index) {
+ return binary_pk_value(row, column_index);
+ }
+ Ok(column_value_as_string(row, column_index, value)?.map(PkValue::Text))
+}
+
+fn is_binary_column(row: &MySqlRow, column_index: usize) -> bool {
+ matches!(
+ row.columns()[column_index].type_info().name(),
+ "BINARY" | "VARBINARY" | "TINYBLOB" | "BLOB" | "MEDIUMBLOB" | "LONGBLOB"
+ )
+}
+
+fn binary_pk_value(row: &MySqlRow, column_index: usize) -> Result<Option<PkValue>, Error> {
+ let column = &row.columns()[column_index];
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "primary key column '{}' (MySQL type '{}'): {e}",
+ column.name(),
+ column.type_info().name()
+ ))
+ })?;
+ Ok(bytes.map(PkValue::Bytes))
+}
+
+fn extract_column_value(row: &MySqlRow, column_index: usize) -> Result<serde_json::Value, Error> {
+ let column = &row.columns()[column_index];
+ let type_name = column.type_info().name();
+
+ let to_err = |e: sqlx::Error| {
+ Error::InvalidRecordValue(format!(
+ "column '{}' (MySQL type '{type_name}'): {e}",
+ column.name()
+ ))
+ };
+
+ match type_name {
+ "BOOLEAN" => {
+ let value: Option<bool> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::Bool)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT" => {
+ let value: Option<i8> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT" => {
+ let value: Option<i16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT" | "INT" => {
+ let value: Option<i32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT" => {
+ let value: Option<i64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT UNSIGNED" => {
+ let value: Option<u8> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT UNSIGNED" => {
+ let value: Option<u16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT UNSIGNED" | "INT UNSIGNED" => {
+ let value: Option<u32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT UNSIGNED" => {
+ let value: Option<u64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "FLOAT" => {
+ let value: Option<f32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as f64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DOUBLE" => {
+ let value: Option<f64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DECIMAL" => {
+ let value: Option<rust_decimal::Decimal> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|d| serde_json::Value::String(d.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIT" => {
+ let value: Option<u64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "YEAR" => {
+ let value: Option<u16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATE" => {
+ let value: Option<NaiveDate> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|d| serde_json::Value::String(d.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TIME" => {
+ let value: Option<NaiveTime> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|t| serde_json::Value::String(t.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATETIME" => {
+ let value: Option<NaiveDateTime> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|dt| serde_json::Value::String(dt.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TIMESTAMP" => {
+ let value: Option<DateTime<Utc>> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|dt| serde_json::Value::String(dt.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT" | "MEDIUMTEXT" | "LONGTEXT" | "ENUM" | "SET" => {
+ let value: Option<String> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BINARY" | "VARBINARY" | "TINYBLOB" | "BLOB" | "MEDIUMBLOB" | "LONGBLOB" => {
+ let value: Option<Vec<u8>> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|bytes| {
+ serde_json::Value::String(
+ base64::engine::general_purpose::STANDARD.encode(&bytes),
+ )
+ })
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "JSON" => {
+ let value: Option<serde_json::Value> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value.unwrap_or(serde_json::Value::Null))
+ }
+ "NULL" => Ok(serde_json::Value::Null),
+ _ => {
+ let column_name = column.name();
+ warn!(
+ "Column '{column_name}' has unrecognized MySQL type '{type_name}', \
+ attempting text extraction"
+ );
+ if let Ok(text) = row.try_get::<Option<String>, _>(column_index) {
+ return Ok(text
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null));
+ }
+ if let Ok(bytes) = row.try_get::<Option<Vec<u8>>, _>(column_index) {
+ return Ok(bytes
+ .map(|b| {
+ serde_json::Value::String(
+ base64::engine::general_purpose::STANDARD.encode(&b),
+ )
+ })
+ .unwrap_or(serde_json::Value::Null));
+ }
+ error!(
+ "Column '{column_name}' has unsupported MySQL type '{type_name}', \
+ returning null"
+ );
+ Ok(serde_json::Value::Null)
Review Comment:
Fixed: the fallback now returns `InvalidRecordValue`, so the batch fails before publish and the cursor doesn't move.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3603878228
##########
core/connectors/sources/mysql_source/Cargo.toml:
##########
@@ -0,0 +1,54 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[package]
+name = "iggy_connector_mysql_source"
+version = "0.1.0"
+description = "Iggy MySQL source connector supporting table polling for message streaming platform"
+edition = "2024"
+license = "Apache-2.0"
+keywords = ["iggy", "messaging", "streaming", "mysql", "polling"]
+categories = ["command-line-utilities", "database", "network-programming"]
+homepage = "https://iggy.apache.org"
+documentation = "https://iggy.apache.org/docs"
+repository = "https://github.com/apache/iggy"
+readme = "README.md"
+publish = false
+
+[package.metadata.cargo-machete]
+ignored = ["dashmap"]
+
+[lib]
+crate-type = ["cdylib", "lib"]
+
+[dependencies]
+async-trait = { workspace = true }
+base64 = { workspace = true }
+chrono = { workspace = true }
Review Comment:
dashmap dep unused in lib.rs. Validate and **Fix**: remove dep + machete ignore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "slbotbm (via GitHub)" <gi...@apache.org>.
slbotbm commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3699476507
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
Review Comment:
`MySqlSource::validate_tracking_column` returns immediately whenever `custom_query` is configured (`lib.rs:349`). Later, `process_row` permits a row without the configured tracking column, leaves `max_offset` as `None`, and still emits the message. A query that omits or aliases the tracking column therefore replays the same result on every poll while never updating state.
Validate each custom-query result contains a non-null, supported tracking value, or fail the table batch before emitting anything. The open-time schema probe did not fully address the earlier [comment](https://github.com/apache/iggy/pull/3568#discussion_r3603820903) because custom queries explicitly bypass that probe.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.connect().await?;
+ self.validate_tracking_column().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ // Skip the pacing delay while draining a backlog
+ if !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ // Idle cycles produce no messages and leave offsets/processed_rows
+ // untouched, so return None to let the runtime skip the state fsync.
+ let persisted_state = if messages.is_empty() {
+ None
+ } else {
+ self.serialize_state(&state)
+ };
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn validate_tracking_column(&self) -> Result<(), Error> {
+ if self.config.custom_query.is_some() {
+ return Ok(());
+ }
+ let pool = self.get_pool()?;
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ continue;
+ }
+
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ if data_type.eq_ignore_ascii_case("json")
+ || column_type.eq_ignore_ascii_case("tinyint(1)")
+ {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type '{column_type}', \
+ which yields no ordered scalar cursor; use an integer, timestamp, or string column."
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+ let offset = format_offset_value(&offset_value);
+ let now = Utc::now();
+
+ let q = query
+ .replace("$table", "e_qualified_identifier(table)?)
+ .replace("$offset", &offset)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now_unix", &now.timestamp().to_string())
+ .replace("$now", &now.to_rfc3339());
+
+ Ok(q)
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[PkValue],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({placeholders})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ // The query and its binds are rebuilt on every attempt because with_retry
+ // calls the closure once per try and each `execute` consumes the query.
+ with_retry(
+ || {
+ let mut prepared = sqlx::query(sqlx::AssertSqlSafe(query.as_str()));
+ for id in ids {
+ prepared = match id {
+ PkValue::Text(text) => prepared.bind(text),
+ PkValue::Bytes(bytes) => prepared.bind(bytes),
+ };
+ }
+ prepared.execute(pool)
+ },
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
Review Comment:
This loops over every configured table regardless of whether `custom_query` contains `$table`. With multiple tables and a static custom query, the identical SQL is executed once per table and its rows are emitted multiple times with different table-derived metadata/message IDs. Validation only rejects `$table` when the table list is empty; it does not reject this ambiguous combination.
Require `$table` when more than one table is configured, or execute a static custom query exactly once.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.connect().await?;
+ self.validate_tracking_column().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ // Skip the pacing delay while draining a backlog
+ if !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ // Idle cycles produce no messages and leave offsets/processed_rows
+ // untouched, so return None to let the runtime skip the state fsync.
+ let persisted_state = if messages.is_empty() {
+ None
+ } else {
+ self.serialize_state(&state)
+ };
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn validate_tracking_column(&self) -> Result<(), Error> {
+ if self.config.custom_query.is_some() {
+ return Ok(());
+ }
+ let pool = self.get_pool()?;
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ continue;
+ }
+
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ if data_type.eq_ignore_ascii_case("json")
+ || column_type.eq_ignore_ascii_case("tinyint(1)")
+ {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type '{column_type}', \
+ which yields no ordered scalar cursor; use an integer, timestamp, or string column."
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+ let offset = format_offset_value(&offset_value);
+ let now = Utc::now();
+
+ let q = query
+ .replace("$table", "e_qualified_identifier(table)?)
+ .replace("$offset", &offset)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now_unix", &now.timestamp().to_string())
+ .replace("$now", &now.to_rfc3339());
+
+ Ok(q)
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[PkValue],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({placeholders})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ // The query and its binds are rebuilt on every attempt because with_retry
+ // calls the closure once per try and each `execute` consumes the query.
+ with_retry(
+ || {
+ let mut prepared = sqlx::query(sqlx::AssertSqlSafe(query.as_str()));
+ for id in ids {
+ prepared = match id {
+ PkValue::Text(text) => prepared.bind(text),
+ PkValue::Bytes(bytes) => prepared.bind(bytes),
+ };
+ }
+ prepared.execute(pool)
+ },
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ // A full batch means the LIMIT was hit, so more rows are waiting.
+ any_batch_full |= batch.messages.len() as u32 >= batch_size;
+ batches.push(batch);
Review Comment:
There is no positive-value validation for `batch_size`. With zero, MySQL returns an empty `LIMIT 0` result, but the full-batch check evaluates `0 >= 0` as true. `poll` then skips its pacing sleep forever because `last_batch_full` remains true, producing a CPU/database hot loop that can never emit a row.
Reject zero during initialization.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.connect().await?;
+ self.validate_tracking_column().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ // Skip the pacing delay while draining a backlog
+ if !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ // Idle cycles produce no messages and leave offsets/processed_rows
+ // untouched, so return None to let the runtime skip the state fsync.
+ let persisted_state = if messages.is_empty() {
+ None
+ } else {
+ self.serialize_state(&state)
+ };
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn validate_tracking_column(&self) -> Result<(), Error> {
+ if self.config.custom_query.is_some() {
+ return Ok(());
+ }
+ let pool = self.get_pool()?;
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ continue;
+ }
+
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ if data_type.eq_ignore_ascii_case("json")
+ || column_type.eq_ignore_ascii_case("tinyint(1)")
+ {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type '{column_type}', \
+ which yields no ordered scalar cursor; use an integer, timestamp, or string column."
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+ let offset = format_offset_value(&offset_value);
+ let now = Utc::now();
+
+ let q = query
+ .replace("$table", "e_qualified_identifier(table)?)
+ .replace("$offset", &offset)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now_unix", &now.timestamp().to_string())
+ .replace("$now", &now.to_rfc3339());
+
+ Ok(q)
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[PkValue],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({placeholders})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ // The query and its binds are rebuilt on every attempt because with_retry
+ // calls the closure once per try and each `execute` consumes the query.
+ with_retry(
+ || {
+ let mut prepared = sqlx::query(sqlx::AssertSqlSafe(query.as_str()));
+ for id in ids {
+ prepared = match id {
+ PkValue::Text(text) => prepared.bind(text),
+ PkValue::Bytes(bytes) => prepared.bind(bytes),
+ };
+ }
+ prepared.execute(pool)
+ },
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ // A full batch means the LIMIT was hit, so more rows are waiting.
+ any_batch_full |= batch.messages.len() as u32 >= batch_size;
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. If the mark/delete call fails
+ // for a table we just skip its messages this cycle instead of queuing them
+ // - `mark_or_delete_processed_rows` already retries transient errors, so a
+ // failure here is permanent and the table's rows stay in MySQL to be
+ // retried next cycle, without blocking the others.
+ //
+ // Doesn't cover the case where mark/delete succeeds but the runtime never
+ // gets to publish the batch (crash, send failure, etc) - those rows are
+ // gone from MySQL with nothing delivered.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ let mark_or_delete = self.config.delete_after_read.unwrap_or(false)
+ || self.config.processed_column.is_some();
+
+ for mut batch in batches {
+ if mark_or_delete && !batch.messages.is_empty() && batch.processed_ids.is_empty() {
+ error!(
+ "Table '{}': mark/delete is configured but no primary keys were extracted \
+ from {} row(s), skipping this cycle so rows are not published without being \
+ marked or deleted (check that '{pk_column}' is projected and scalar)",
+ batch.table,
+ batch.messages.len()
+ );
+ continue;
+ }
+
+ if !batch.processed_ids.is_empty()
+ && let Err(error) = self
+ .mark_or_delete_processed_rows(
+ pool,
+ &batch.table,
+ pk_column,
+ &batch.processed_ids,
+ )
+ .await
+ {
+ error!(
+ "Failed to mark or delete processed rows for table '{}', skipping this cycle: {error}",
+ batch.table
+ );
+ continue;
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ state_updates.push((batch.table, offset));
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
+ }
+ }
+
+ self.last_batch_full
+ .store(any_batch_full, Ordering::Relaxed);
+ Ok(messages)
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, Error> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.substitute_query_params(custom_query, table, last_offset, batch_size)?
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ for row in rows {
+ let processed = self.process_row(&row, &table_config).map_err(|e| {
+ error!(
+ "Failed to decode row in table '{table}' (pk {}): {e}",
+ pk_for_log(&row, table_config.pk_column)
+ );
+ e
+ })?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+ if let Some(offset) = processed.max_offset {
+ batch.max_offset = Some(offset);
+ }
+
+ batch.messages.push(processed.message);
+ }
+
+ Ok(batch)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<PkValue> = None;
+ let mut max_offset: Option<String> = None;
+ let mut extracted_payload: Option<Vec<u8>> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ if name == config.payload_col {
+ extracted_payload =
+ Some(self.extract_payload_column(row, i, config.payload_format)?);
+ }
+ if name == config.tracking_column {
+ max_offset = value_as_string(&extract_column_value(row, i)?);
+ }
+ if name == config.pk_column {
+ row_pk = extract_pk_value(row, i)?;
+ }
+ }
+ }
+
+ if extracted_payload.is_none() {
+ let mut data = serde_json::Map::new();
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ let column_name = if config.snake_case_columns {
+ to_snake_case(name)
+ } else {
+ name.to_string()
+ };
+ let value = extract_column_value(row, i)?;
+ if name == config.tracking_column {
+ max_offset = value_as_string(&value);
+ }
+ if name == config.pk_column {
+ row_pk = extract_pk_value(row, i)?;
+ }
+ data.insert(column_name, value);
+ }
+
+ extracted_payload = Some(if config.include_metadata {
+ let record = DatabaseRecord {
+ table_name: config.table.to_string(),
+ operation_type: "SELECT".to_string(),
+ timestamp: Utc::now(),
+ data: serde_json::Value::Object(data),
+ old_data: None,
+ };
+ simd_json::to_vec(&record).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': failed to serialize row to JSON: {e}",
+ config.table
+ ))
+ })?
+ } else {
+ simd_json::to_vec(&data).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "table '{}': failed to serialize row to JSON: {e}",
+ config.table
+ ))
+ })?
+ });
+ }
+
+ // Both paths above always assign extracted_payload before reaching here.
+ Ok(build_processed_row(
+ config.table,
+ extracted_payload.unwrap(),
+ max_offset,
+ row_pk,
+ ))
+ }
+}
+
+/// Type-faithful primary-key extraction. Binary columns are taken as raw bytes
+/// rather than routed through the base64 path in `extract_column_value`, so the
+/// value can be bound back into `WHERE pk IN (...)` and match the row. Everything
+/// else reuses the scalar string form, which MySQL implicitly converts. Returns
+/// None only for a NULL pk, which a real primary key cannot be.
+fn extract_pk_value(row: &MySqlRow, column_index: usize) -> Result<Option<PkValue>, Error> {
+ let column = &row.columns()[column_index];
+ let type_name = column.type_info().name();
+ match type_name {
+ "BINARY" | "VARBINARY" | "TINYBLOB" | "BLOB" | "MEDIUMBLOB" | "LONGBLOB" => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "primary key column '{}' (MySQL type '{type_name}'): {e}",
+ column.name()
+ ))
+ })?;
+ Ok(bytes.map(PkValue::Bytes))
+ }
+ _ => Ok(value_as_string(&extract_column_value(row, column_index)?).map(PkValue::Text)),
+ }
+}
+
+fn extract_column_value(row: &MySqlRow, column_index: usize) -> Result<serde_json::Value, Error> {
+ let column = &row.columns()[column_index];
+ let type_name = column.type_info().name();
+
+ let to_err = |e: sqlx::Error| {
+ Error::InvalidRecordValue(format!(
+ "column '{}' (MySQL type '{type_name}'): {e}",
+ column.name()
+ ))
+ };
+
+ match type_name {
+ "BOOLEAN" => {
+ let value: Option<bool> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::Bool)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT" => {
+ let value: Option<i8> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT" => {
+ let value: Option<i16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT" | "INT" => {
+ let value: Option<i32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT" => {
+ let value: Option<i64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT UNSIGNED" => {
+ let value: Option<u8> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT UNSIGNED" => {
+ let value: Option<u16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT UNSIGNED" | "INT UNSIGNED" => {
+ let value: Option<u32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT UNSIGNED" => {
+ let value: Option<u64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "FLOAT" => {
+ let value: Option<f32> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as f64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DOUBLE" => {
+ let value: Option<f64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DECIMAL" => {
+ let value: Option<rust_decimal::Decimal> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|d| serde_json::Value::String(d.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIT" => {
+ let value: Option<u64> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "YEAR" => {
+ let value: Option<u16> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATE" => {
+ let value: Option<NaiveDate> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|d| serde_json::Value::String(d.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TIME" => {
+ let value: Option<NaiveTime> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|t| serde_json::Value::String(t.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATETIME" => {
+ let value: Option<NaiveDateTime> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|dt| serde_json::Value::String(dt.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TIMESTAMP" => {
+ let value: Option<DateTime<Utc>> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|dt| serde_json::Value::String(dt.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT" | "MEDIUMTEXT" | "LONGTEXT" | "ENUM" | "SET" => {
+ let value: Option<String> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BINARY" | "VARBINARY" | "TINYBLOB" | "BLOB" | "MEDIUMBLOB" | "LONGBLOB" => {
+ let value: Option<Vec<u8>> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value
+ .map(|bytes| {
+ serde_json::Value::String(
+ base64::engine::general_purpose::STANDARD.encode(&bytes),
+ )
+ })
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "JSON" => {
+ let value: Option<serde_json::Value> = row.try_get(column_index).map_err(to_err)?;
+ Ok(value.unwrap_or(serde_json::Value::Null))
+ }
+ "NULL" => Ok(serde_json::Value::Null),
+ _ => {
+ let column_name = column.name();
+ warn!(
+ "Column '{column_name}' has unrecognized MySQL type '{type_name}', \
+ attempting text extraction"
+ );
+ if let Ok(text) = row.try_get::<Option<String>, _>(column_index) {
+ return Ok(text
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null));
+ }
+ if let Ok(bytes) = row.try_get::<Option<Vec<u8>>, _>(column_index) {
+ return Ok(bytes
+ .map(|b| {
+ serde_json::Value::String(
+ base64::engine::general_purpose::STANDARD.encode(&b),
+ )
+ })
+ .unwrap_or(serde_json::Value::Null));
+ }
+ error!(
+ "Column '{column_name}' has unsupported MySQL type '{type_name}', \
+ returning null"
+ );
+ Ok(serde_json::Value::Null)
+ }
+ }
+}
+
+/// Best-effort primary-key value for diagnostics when a row fails to decode.
+/// Falls back to "unknown" if the pk column is absent or itself fails to extract,
+/// so logging a decode failure never masks it with a second error.
+fn pk_for_log(row: &MySqlRow, pk_column: &str) -> String {
+ row.columns()
+ .iter()
+ .position(|column| column.name() == pk_column)
+ .and_then(|index| extract_column_value(row, index).ok())
+ .and_then(|value| value_as_string(&value))
+ .unwrap_or_else(|| "unknown".to_string())
+}
+
+fn value_as_string(value: &serde_json::Value) -> Option<String> {
+ match value {
+ serde_json::Value::String(s) => Some(s.clone()),
+ serde_json::Value::Number(n) => Some(n.to_string()),
+ _ => None,
+ }
+}
+
+fn build_processed_row(
+ table: &str,
+ payload: Vec<u8>,
+ max_offset: Option<String>,
+ row_pk: Option<PkValue>,
+) -> ProcessedRow {
Review Comment:
`build_processed_row` uses `Utc::now().timestamp_millis()` for both `timestamp` and `origin_timestamp`. Iggy message timestamps are microseconds; the core initializes them with `IggyTimestamp::now().as_micros()`, and the InfluxDB sources use `timestamp_micros()`. The current value is 1,000 times too small and will be interpreted as a date near January 1970.
Use `timestamp_micros()` or leave the fields unset so the normal Iggy timestamping path applies. Add an assertion for timestamp units to the integration coverage.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.connect().await?;
+ self.validate_tracking_column().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ // Skip the pacing delay while draining a backlog
+ if !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ // Idle cycles produce no messages and leave offsets/processed_rows
+ // untouched, so return None to let the runtime skip the state fsync.
+ let persisted_state = if messages.is_empty() {
+ None
+ } else {
+ self.serialize_state(&state)
+ };
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn validate_tracking_column(&self) -> Result<(), Error> {
+ if self.config.custom_query.is_some() {
+ return Ok(());
+ }
+ let pool = self.get_pool()?;
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ continue;
+ }
+
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ if data_type.eq_ignore_ascii_case("json")
+ || column_type.eq_ignore_ascii_case("tinyint(1)")
+ {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type '{column_type}', \
+ which yields no ordered scalar cursor; use an integer, timestamp, or string column."
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+ let offset = format_offset_value(&offset_value);
+ let now = Utc::now();
+
+ let q = query
+ .replace("$table", "e_qualified_identifier(table)?)
+ .replace("$offset", &offset)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now_unix", &now.timestamp().to_string())
+ .replace("$now", &now.to_rfc3339());
+
+ Ok(q)
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[PkValue],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({placeholders})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ // The query and its binds are rebuilt on every attempt because with_retry
+ // calls the closure once per try and each `execute` consumes the query.
+ with_retry(
+ || {
+ let mut prepared = sqlx::query(sqlx::AssertSqlSafe(query.as_str()));
+ for id in ids {
+ prepared = match id {
+ PkValue::Text(text) => prepared.bind(text),
+ PkValue::Bytes(bytes) => prepared.bind(bytes),
+ };
+ }
+ prepared.execute(pool)
+ },
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ // A full batch means the LIMIT was hit, so more rows are waiting.
+ any_batch_full |= batch.messages.len() as u32 >= batch_size;
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. If the mark/delete call fails
+ // for a table we just skip its messages this cycle instead of queuing them
+ // - `mark_or_delete_processed_rows` already retries transient errors, so a
+ // failure here is permanent and the table's rows stay in MySQL to be
+ // retried next cycle, without blocking the others.
+ //
+ // Doesn't cover the case where mark/delete succeeds but the runtime never
+ // gets to publish the batch (crash, send failure, etc) - those rows are
+ // gone from MySQL with nothing delivered.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ let mark_or_delete = self.config.delete_after_read.unwrap_or(false)
+ || self.config.processed_column.is_some();
+
+ for mut batch in batches {
+ if mark_or_delete && !batch.messages.is_empty() && batch.processed_ids.is_empty() {
+ error!(
+ "Table '{}': mark/delete is configured but no primary keys were extracted \
+ from {} row(s), skipping this cycle so rows are not published without being \
+ marked or deleted (check that '{pk_column}' is projected and scalar)",
+ batch.table,
+ batch.messages.len()
+ );
+ continue;
+ }
+
+ if !batch.processed_ids.is_empty()
+ && let Err(error) = self
+ .mark_or_delete_processed_rows(
+ pool,
+ &batch.table,
+ pk_column,
+ &batch.processed_ids,
+ )
+ .await
+ {
+ error!(
+ "Failed to mark or delete processed rows for table '{}', skipping this cycle: {error}",
+ batch.table
+ );
+ continue;
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ state_updates.push((batch.table, offset));
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
+ }
+ }
+
+ self.last_batch_full
+ .store(any_batch_full, Ordering::Relaxed);
+ Ok(messages)
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, Error> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.substitute_query_params(custom_query, table, last_offset, batch_size)?
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ for row in rows {
+ let processed = self.process_row(&row, &table_config).map_err(|e| {
+ error!(
+ "Failed to decode row in table '{table}' (pk {}): {e}",
+ pk_for_log(&row, table_config.pk_column)
+ );
+ e
+ })?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+ if let Some(offset) = processed.max_offset {
+ batch.max_offset = Some(offset);
+ }
+
+ batch.messages.push(processed.message);
+ }
+
+ Ok(batch)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<PkValue> = None;
+ let mut max_offset: Option<String> = None;
+ let mut extracted_payload: Option<Vec<u8>> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
Review Comment:
If the configured column is absent from a custom query result or is misspelled, `extracted_payload` stays `None`; the fallback at `lib.rs:864` serializes the entire row as JSON. The batch schema still comes from `payload_format`, so a `bytea` or `text` source can silently publish JSON bytes, while `json_direct` publishes the wrong JSON shape.
Treat a missing configured payload column as an error. Falling back is only valid when `payload_column` was not configured.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3706999756
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1721 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.connect().await?;
+ self.validate_tracking_column().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ // Skip the pacing delay while draining a backlog
+ if !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ // Idle cycles produce no messages and leave offsets/processed_rows
+ // untouched, so return None to let the runtime skip the state fsync.
+ let persisted_state = if messages.is_empty() {
+ None
+ } else {
+ self.serialize_state(&state)
+ };
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn validate_tracking_column(&self) -> Result<(), Error> {
+ if self.config.custom_query.is_some() {
+ return Ok(());
+ }
+ let pool = self.get_pool()?;
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ continue;
+ }
+
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ if data_type.eq_ignore_ascii_case("json")
+ || column_type.eq_ignore_ascii_case("tinyint(1)")
+ {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type '{column_type}', \
+ which yields no ordered scalar cursor; use an integer, timestamp, or string column."
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+ let offset = format_offset_value(&offset_value);
+ let now = Utc::now();
+
+ let q = query
+ .replace("$table", "e_qualified_identifier(table)?)
+ .replace("$offset", &offset)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now_unix", &now.timestamp().to_string())
+ .replace("$now", &now.to_rfc3339());
+
+ Ok(q)
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[PkValue],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({placeholders})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ // The query and its binds are rebuilt on every attempt because with_retry
+ // calls the closure once per try and each `execute` consumes the query.
+ with_retry(
+ || {
+ let mut prepared = sqlx::query(sqlx::AssertSqlSafe(query.as_str()));
+ for id in ids {
+ prepared = match id {
+ PkValue::Text(text) => prepared.bind(text),
+ PkValue::Bytes(bytes) => prepared.bind(bytes),
+ };
+ }
+ prepared.execute(pool)
+ },
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ // A full batch means the LIMIT was hit, so more rows are waiting.
+ any_batch_full |= batch.messages.len() as u32 >= batch_size;
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. If the mark/delete call fails
+ // for a table we just skip its messages this cycle instead of queuing them
+ // - `mark_or_delete_processed_rows` already retries transient errors, so a
+ // failure here is permanent and the table's rows stay in MySQL to be
+ // retried next cycle, without blocking the others.
+ //
+ // Doesn't cover the case where mark/delete succeeds but the runtime never
+ // gets to publish the batch (crash, send failure, etc) - those rows are
+ // gone from MySQL with nothing delivered.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ let mark_or_delete = self.config.delete_after_read.unwrap_or(false)
+ || self.config.processed_column.is_some();
+
+ for mut batch in batches {
+ if mark_or_delete && !batch.messages.is_empty() && batch.processed_ids.is_empty() {
+ error!(
+ "Table '{}': mark/delete is configured but no primary keys were extracted \
+ from {} row(s), skipping this cycle so rows are not published without being \
+ marked or deleted (check that '{pk_column}' is projected and scalar)",
+ batch.table,
+ batch.messages.len()
+ );
+ continue;
+ }
+
+ if !batch.processed_ids.is_empty()
+ && let Err(error) = self
+ .mark_or_delete_processed_rows(
+ pool,
+ &batch.table,
+ pk_column,
+ &batch.processed_ids,
+ )
+ .await
+ {
+ error!(
+ "Failed to mark or delete processed rows for table '{}', skipping this cycle: {error}",
+ batch.table
+ );
+ continue;
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ state_updates.push((batch.table, offset));
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
+ }
+ }
+
+ self.last_batch_full
+ .store(any_batch_full, Ordering::Relaxed);
+ Ok(messages)
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, Error> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.substitute_query_params(custom_query, table, last_offset, batch_size)?
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ for row in rows {
+ let processed = self.process_row(&row, &table_config).map_err(|e| {
+ error!(
+ "Failed to decode row in table '{table}' (pk {}): {e}",
+ pk_for_log(&row, table_config.pk_column)
+ );
+ e
+ })?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+ if let Some(offset) = processed.max_offset {
+ batch.max_offset = Some(offset);
+ }
+
+ batch.messages.push(processed.message);
+ }
+
+ Ok(batch)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<PkValue> = None;
+ let mut max_offset: Option<String> = None;
+ let mut extracted_payload: Option<Vec<u8>> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
Review Comment:
`!config.payload_col.is_empty()` block now handles and fails that table for that cycle (per table basis), when `extracted_payload.is_none()` even when `payload_col` is some. Also added integration test, where `payload_col` is set but is not present in the table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3616956507
##########
core/connectors/sources/mysql_source/README.md:
##########
@@ -0,0 +1,424 @@
+# MySQL Source Connector
+
+The MySQL source connector fetches data from MySQL databases and streams it to Iggy topics. It supports incremental table polling with flexible payload extraction.
+
+> **Note:** Only polling mode is available. Binlog CDC support is planned for a future release.
+
+## Features
+
+- **Table Polling**: Incrementally fetch data from MySQL tables using a tracking column
+- **Flexible Payload Extraction**: Extract BLOB, TEXT, or JSON columns directly as payload
+- **Custom Queries**: Use custom SQL queries with parameter substitution
+- **Delete After Read**: Automatically delete rows after processing
+- **Mark as Processed**: Mark rows as processed using a boolean column
+- **Multiple Tables**: Monitor multiple tables simultaneously
+- **Batch Processing**: Fetch data in configurable batch sizes
+- **Offset Tracking**: Resume incremental polling from the last processed tracking value (see [Delivery semantics](#delivery-semantics))
+
+## Configuration
+
+```toml
+[plugin_config]
+connection_string = "mysql://user:pass@localhost:3306/database"
+tables = ["users", "orders"]
+poll_interval = "1s"
+batch_size = 1000
+tracking_column = "id"
+initial_offset = "0"
+max_connections = 10
+snake_case_columns = false
+include_metadata = true
+
+# Payload extraction (optional)
+payload_column = "payload"
+payload_format = "bytea"
+
+# Delete/mark processed (optional)
+delete_after_read = false
+processed_column = "is_processed"
+primary_key_column = "id"
+
+# Custom query (optional)
+custom_query = "SELECT * FROM $table WHERE id > $offset ORDER BY id LIMIT $limit"
+```
+
+## Configuration Options
+
+| Option | Type | Default | Description |
+| ------ | ---- | ------- | ----------- |
+| `connection_string` | string | required | MySQL connection string (`mysql://user:pass@host:3306/db`) |
+| `tables` | array | required | List of tables to monitor |
+| `poll_interval` | string | `10s` | How often to poll (e.g., `1s`, `5m`) |
+| `batch_size` | u32 | `1000` | Max rows per poll |
+| `tracking_column` | string | `id` | Column for incremental polling; must be unique and monotonically increasing (see [Tracking Column Requirements](#tracking-column-requirements)) |
+| `initial_offset` | string | none | Starting value for tracking column |
+| `max_connections` | u32 | `10` | Max database connections |
+| `snake_case_columns` | bool | `false` | Convert column names to snake_case |
+| `include_metadata` | bool | `true` | Wrap results with metadata envelope |
+| `payload_column` | string | none | Column to extract directly as payload |
+| `payload_format` | string | `json` (invalid if `payload_column` is set — `payload_format` is required in that case) | Format of payload_column: `bytea`, `text`, or `json_direct` |
+| `delete_after_read` | bool | `false` | Delete rows after reading |
+| `processed_column` | string | none | Boolean column to mark as processed |
+| `primary_key_column` | string | tracking_column | PK for delete/mark operations |
+| `custom_query` | string | none | Custom SQL with parameter substitution |
+| `verbose_logging` | bool | `false` | Log at info level instead of debug |
+| `max_retries` | u32 | `3` | Retries after the initial attempt for transient errors (3 = 4 total attempts) |
+| `retry_delay` | string | `1s` | Base delay between retries (e.g., `500ms`, `2s`) |
+
+## Tracking Column Requirements
+
+Polling is **insert-only**. Each poll runs roughly `SELECT ... WHERE tracking_column > last_offset ORDER BY tracking_column ASC LIMIT batch_size` and stores the largest tracking value it saw as the next offset. For this to be lossless the **`tracking_column` must be unique and monotonically increasing** — an auto-increment primary key is the canonical choice.
+
+The requirement is not cosmetic: if more than `batch_size` rows share the same tracking value, only the first `batch_size` are read and the next poll's `>` filter skips the rest of that value permanently. The same skip applies to `processed_column` and `delete_after_read`, because the tracking filter runs before those.
+
+For this reason a mutable, low-resolution column such as `updated_at` is **not** a safe tracking column: many rows commonly share the same second, and rows updated in place are never re-read. Capturing updates to existing rows is out of scope for polling.
+
+## Output Modes
+
+### JSON Mode (Default)
+
+When `payload_column` is not set, each row is wrapped in a `DatabaseRecord` JSON structure:
+
+```json
+{
+ "table_name": "users",
+ "operation_type": "SELECT",
+ "timestamp": "2024-01-15T10:30:00Z",
+ "data": {
+ "id": 123,
+ "name": "John Doe",
+ "email": "john@example.com"
+ },
+ "old_data": null
+}
+```
+
+The stream config should use `schema = "json"`.
+
+When `include_metadata = false`, the envelope is omitted and the row columns are serialized directly:
+
+```json
+{
+ "id": 123,
+ "name": "John Doe",
+ "email": "john@example.com"
+}
+```
+
+### Payload Column Extraction
+
+When `payload_column` is set, the connector extracts that column directly as the Iggy message payload. The `payload_format` option determines how the column is read:
+
+| Format | Column Type | Schema | Description |
+| ------ | ----------- | ------ | ----------- |
+| `bytea` / `raw` | `BLOB`, `BINARY`, `VARBINARY` | `raw` | Raw bytes passthrough |
+| `text` | `TEXT`, `VARCHAR` | `text` | UTF-8 text |
+| `json_direct` / `jsonb` | `JSON` | `json` | JSON object serialized to bytes |
+
+## Payload Format Examples
+
+### BLOB (Raw Bytes)
+
+Extract raw bytes from a BLOB column:
+
+```sql
+CREATE TABLE message_queue (
+ id INT AUTO_INCREMENT PRIMARY KEY,
+ payload BLOB NOT NULL
+);
+```
+
+```toml
+[[streams]]
+stream = "messages"
+topic = "queue"
+schema = "raw"
+batch_length = 100
+
+[plugin_config]
+tables = ["message_queue"]
+tracking_column = "id"
+payload_column = "payload"
+payload_format = "bytea"
+```
+
+### TEXT
+
+Extract text from a TEXT column:
+
+```sql
+CREATE TABLE logs (
+ id INT AUTO_INCREMENT PRIMARY KEY,
+ message TEXT NOT NULL
+);
+```
+
+```toml
+[[streams]]
+stream = "logs"
+topic = "app_logs"
+schema = "text"
+batch_length = 100
+
+[plugin_config]
+tables = ["logs"]
+tracking_column = "id"
+payload_column = "message"
+payload_format = "text"
+```
+
+### JSON (Direct)
+
+Extract a JSON column directly as JSON payload (without `DatabaseRecord` wrapper):
+
+```sql
+CREATE TABLE events (
+ id INT AUTO_INCREMENT PRIMARY KEY,
+ data JSON NOT NULL
+);
+```
+
+```toml
+[[streams]]
+stream = "events"
+topic = "user_events"
+schema = "json"
+batch_length = 100
+
+[plugin_config]
+tables = ["events"]
+tracking_column = "id"
+payload_column = "data"
+payload_format = "json_direct"
+```
+
+## Custom Query Parameters
+
+When using `custom_query`, these placeholders are available:
+
+| Placeholder | Replaced With |
+| ----------- | ------------- |
+| `$table` | Current table name |
+| `$offset` | Last processed offset (or `initial_offset`) |
+| `$limit` | `batch_size` value |
+| `$now` | Current UTC timestamp (RFC3339) |
+| `$now_unix` | Current Unix timestamp (seconds) |
+
+If the query uses `$offset`, it must be ordered by the tracking column in ascending order (`ORDER BY <tracking_column>` — MySQL sorts ascending by default, so `ASC` doesn't need to be spelled out). The connector takes the tracking value of the *last row returned* as the next `$offset`; without ascending order, that's not guaranteed to be the max, so rows can be skipped or re-emitted on the next poll.
+
+The column you order by is your effective tracking column, so the same [Tracking Column Requirements](#tracking-column-requirements) apply: it must be unique and monotonically increasing, or rows sharing a value can be skipped across a batch boundary.
+
+Example:
+
+```sql
+SELECT * FROM $table
+WHERE id > $offset
+ AND (scheduled_at IS NULL OR scheduled_at <= '$now')
+ORDER BY id
+LIMIT $limit
+```
+
+## Delete After Read / Mark as Processed
+
+### Delete After Read
+
+Deletes rows from the source table after successful processing. At-most-once — see [Delivery semantics](#delivery-semantics) below.
+
+```toml
+[plugin_config]
+delete_after_read = true
+primary_key_column = "id"
+```
+
+### Mark as Processed
+
+Updates a boolean column instead of deleting. At-most-once — see [Delivery semantics](#delivery-semantics) below.
+
+```toml
+[plugin_config]
+processed_column = "is_processed"
+primary_key_column = "id"
+```
+
+Your table needs the boolean column:
+
+```sql
+ALTER TABLE users ADD COLUMN is_processed BOOLEAN DEFAULT false;
+```
+
+When `processed_column` is set, the connector automatically adds a `WHERE is_processed = FALSE` filter to the polling query, so only unprocessed rows are fetched.
+
+### Delivery semantics
+
+Both `delete_after_read` and `processed_column` mutate MySQL during the poll, before the batch is actually sent to Iggy. If the connector crashes or the send fails in that window, those rows are gone or marked processed anyway — they won't be retried. So both options are at-most-once, not at-least-once.
Review Comment:
it's a mutually-exclusive if / else if exactly one mutation per row per pass. `delete_after_read` = true deletes the row, otherwise `processed_column` marks it. `delete_after_read` takes precedence when both are set (there's also a startup warning for that case). Never both. Updated the doc too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3529750402
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ let query = if self.config.delete_after_read.unwrap_or(false) {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})")
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ )
+ } else {
+ // Offset-tracking only: nothing to mark or delete.
+ return Ok(());
+ };
+
+ with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).execute(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ batches.push(batch);
+ }
+ Err(error) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ // Phase 2: commit each table independently. A table's messages are emitted
+ // only after its rows are marked or deleted, so a failure here can never
+ // leave a table marked-but-unpublished. `mark_or_delete_processed_rows`
+ // retries transient errors; a permanent failure isolates that one table
+ // (its rows stay in MySQL and are retried next cycle) while the others
+ // still flow.
+ let mut messages = Vec::new();
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if !batch.processed_ids.is_empty()
+ && let Err(error) = self
+ .mark_or_delete_processed_rows(
+ pool,
+ &batch.table,
+ pk_column,
+ &batch.processed_ids,
+ )
+ .await
+ {
+ error!(
+ "Failed to mark or delete processed rows for table '{}', skipping this cycle: {error}",
+ batch.table
+ );
+ continue;
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ state_updates.push((batch.table, offset));
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
+ }
+ state.last_poll_time = Utc::now();
+ }
+
+ Ok(messages)
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, Error> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.validate_custom_query(custom_query)?;
+ self.substitute_query_params(custom_query, table, last_offset, batch_size)
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut batch = TableBatch {
+ table: table.to_string(),
+ messages: Vec::with_capacity(rows.len()),
+ processed_ids: Vec::new(),
+ max_offset: None,
+ };
+ for row in rows {
+ let processed = self.process_row(&row, &table_config)?;
+
+ if let Some(pk) = processed.row_pk {
+ batch.processed_ids.push(pk);
+ }
+ if let Some(offset) = processed.max_offset {
+ batch.max_offset = Some(offset);
Review Comment:
mysql sorts in ASC by default for queries containing `ORDER_BY`, but will make it documented in the readme.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3523743284
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1418 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+/// One table's fully processed but not-yet-committed work. Built in the
+/// side-effect-free first phase of `poll_tables`, then marked/deleted and
+/// published in the second phase so a table's messages are emitted only once
+/// its rows are marked.
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<String>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
Review Comment:
but this might break a normal case also when query contains `$offset` and `last_offset` was there. I think checking both `initial_offset` and `last_offset` and only throw error when both are not present.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3568:
URL: https://github.com/apache/iggy/pull/3568#issuecomment-4804075665
<!-- iggy-pr-triage-welcome -->
Thanks for the PR. It is labeled `S-waiting-on-review` and queued for review.
Slash commands (own line, regular comment) move it around the queue:
- `/ready` - back to `S-waiting-on-review` after addressing feedback
- `/author` - flip to `S-waiting-on-author` while you finish changes
- `/request-review @user-or-team` - request a reviewer
See [CONTRIBUTING.md](https://github.com/apache/iggy/blob/master/CONTRIBUTING.md#pr-triage-commands) for details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "atharvalade (via GitHub)" <gi...@apache.org>.
atharvalade commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r3478146901
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1358 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ if self.config.delete_after_read.unwrap_or(false) {
+ let delete_query =
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})");
+
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+
+ sqlx::query(sqlx::AssertSqlSafe(delete_query))
+ .execute(pool)
+ .await
+ .map_err(|e| {
+ error!("Failed to delete processed rows: {e}");
+ Error::InvalidRecord
+ })?;
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ let update_query = format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ );
+
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+
+ sqlx::query(sqlx::AssertSqlSafe(update_query))
+ .execute(pool)
+ .await
+ .map_err(|e| {
+ error!("Failed to mark rows as processed: {e}");
+ Error::InvalidRecord
+ })?;
+ }
+
+ Ok(())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+ let mut messages = Vec::new();
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Collect state updates to apply after processing
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for table in &self.config.tables {
+ let table_config = RowProcessingConfig {
+ table,
+ ..row_config
+ };
+
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.validate_custom_query(custom_query)?;
+ self.substitute_query_params(custom_query, table, &last_offset, batch_size)
+ } else {
+ self.build_polling_query(table, tracking_column, &last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut max_offset: Option<String> = None;
+ let mut processed_ids: Vec<String> = Vec::new();
+
+ let mut count_per_table = 0;
+ for row in rows {
+ let processed = self.process_row(&row, &table_config)?;
+
+ if let Some(pk) = processed.row_pk {
+ processed_ids.push(pk);
+ }
+ if let Some(offset) = processed.max_offset {
+ max_offset = Some(offset);
+ }
+
+ messages.push(processed.message);
+ count_per_table += 1;
+ total_processed += 1;
+ }
+
+ // Database I/O without holding the lock
+ if !processed_ids.is_empty() {
+ self.mark_or_delete_processed_rows(pool, table, pk_column, &processed_ids)
+ .await?;
+ }
+
+ // Collect offset update for later
+ if let Some(offset) = max_offset {
+ state_updates.push((table.clone(), offset));
+ }
+
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", count_per_table);
+ } else {
+ debug!("Fetched {} rows from table '{table}'", count_per_table);
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
+ }
+ state.last_poll_time = Utc::now();
+ }
+
+ Ok(messages)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<String> = None;
+ let mut max_offset: Option<String> = None;
+ let mut extracted_payload: Option<Vec<u8>> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ if name == config.payload_col {
+ extracted_payload =
+ Some(self.extract_payload_column(row, i, config.payload_format)?);
+ }
+ if name == config.tracking_column {
+ max_offset = value_as_string(&extract_column_value(row, i)?);
+ }
+ if name == config.pk_column {
+ row_pk = value_as_string(&extract_column_value(row, i)?);
+ }
+ }
+ }
+
+ if extracted_payload.is_none() {
+ let mut data = serde_json::Map::new();
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ let column_name = if config.snake_case_columns {
+ to_snake_case(name)
+ } else {
+ name.to_string()
+ };
+ let value = extract_column_value(row, i)?;
+ if name == config.tracking_column {
+ max_offset = value_as_string(&value);
+ }
+ if name == config.pk_column {
+ row_pk = value_as_string(&value);
+ }
+ data.insert(column_name, value);
+ }
+
+ extracted_payload = Some(if config.include_metadata {
+ let record = DatabaseRecord {
+ table_name: config.table.to_string(),
+ operation_type: "SELECT".to_string(),
+ timestamp: Utc::now(),
+ data: serde_json::Value::Object(data),
+ old_data: None,
+ };
+ simd_json::to_vec(&record).map_err(|_| Error::InvalidRecord)?
+ } else {
+ simd_json::to_vec(&data).map_err(|_| Error::InvalidRecord)?
+ });
+ }
+
+ // Both paths above always assign extracted_payload before reaching here.
+ Ok(build_processed_row(
+ extracted_payload.unwrap(),
+ max_offset,
+ row_pk,
+ ))
+ }
+}
+
+fn extract_column_value(row: &MySqlRow, column_index: usize) -> Result<serde_json::Value, Error> {
+ let column = &row.columns()[column_index];
+ let type_name = column.type_info().name();
+
+ match type_name {
+ "BOOLEAN" => {
+ let value: Option<bool> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::Bool)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT" => {
+ let value: Option<i8> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT" => {
+ let value: Option<i16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT" | "INT" => {
+ let value: Option<i32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT" => {
+ let value: Option<i64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT UNSIGNED" => {
+ let value: Option<u8> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT UNSIGNED" => {
+ let value: Option<u16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT UNSIGNED" | "INT UNSIGNED" => {
+ let value: Option<u32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT UNSIGNED" => {
+ let value: Option<u64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "FLOAT" => {
+ let value: Option<f32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as f64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DOUBLE" => {
+ let value: Option<f64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DECIMAL" => {
+ let value: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .and_then(|s| s.parse::<f64>().ok())
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
Review Comment:
Parsing DECIMAL to f64 rounds values beyond ~15 significant digits (e.g. `99999999999999999` becomes `100000000000000000`). This is silent data corruption for large integers or high-precision decimals. Keeping it as a JSON string would preserve the exact value without downstream breakage.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1358 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ if self.config.delete_after_read.unwrap_or(false) {
+ let delete_query =
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})");
+
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+
+ sqlx::query(sqlx::AssertSqlSafe(delete_query))
+ .execute(pool)
+ .await
+ .map_err(|e| {
+ error!("Failed to delete processed rows: {e}");
+ Error::InvalidRecord
+ })?;
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ let update_query = format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ );
+
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+
+ sqlx::query(sqlx::AssertSqlSafe(update_query))
+ .execute(pool)
+ .await
+ .map_err(|e| {
+ error!("Failed to mark rows as processed: {e}");
+ Error::InvalidRecord
+ })?;
+ }
+
+ Ok(())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+ let mut messages = Vec::new();
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Collect state updates to apply after processing
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for table in &self.config.tables {
+ let table_config = RowProcessingConfig {
+ table,
+ ..row_config
+ };
+
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.validate_custom_query(custom_query)?;
+ self.substitute_query_params(custom_query, table, &last_offset, batch_size)
+ } else {
+ self.build_polling_query(table, tracking_column, &last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut max_offset: Option<String> = None;
+ let mut processed_ids: Vec<String> = Vec::new();
+
+ let mut count_per_table = 0;
+ for row in rows {
+ let processed = self.process_row(&row, &table_config)?;
+
+ if let Some(pk) = processed.row_pk {
+ processed_ids.push(pk);
+ }
+ if let Some(offset) = processed.max_offset {
+ max_offset = Some(offset);
+ }
+
+ messages.push(processed.message);
+ count_per_table += 1;
+ total_processed += 1;
+ }
+
+ // Database I/O without holding the lock
+ if !processed_ids.is_empty() {
+ self.mark_or_delete_processed_rows(pool, table, pk_column, &processed_ids)
+ .await?;
+ }
+
+ // Collect offset update for later
+ if let Some(offset) = max_offset {
+ state_updates.push((table.clone(), offset));
+ }
+
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", count_per_table);
+ } else {
+ debug!("Fetched {} rows from table '{table}'", count_per_table);
+ }
+ }
+
+ // Apply all state updates with a single lock acquisition
+ {
+ let mut state = self.state.lock().await;
+ state.processed_rows += total_processed;
+ for (table, offset) in state_updates {
+ state.tracking_offsets.insert(table, offset);
+ }
+ state.last_poll_time = Utc::now();
+ }
+
+ Ok(messages)
+ }
+
+ fn process_row(
+ &self,
+ row: &MySqlRow,
+ config: &RowProcessingConfig,
+ ) -> Result<ProcessedRow, Error> {
+ let mut row_pk: Option<String> = None;
+ let mut max_offset: Option<String> = None;
+ let mut extracted_payload: Option<Vec<u8>> = None;
+
+ // Payload column set: only extract it plus tracking/pk columns.
+ // Avoids extract_column_value on every other column since the data map
+ // built below would be discarded anyway.
+ if !config.payload_col.is_empty() {
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ if name == config.payload_col {
+ extracted_payload =
+ Some(self.extract_payload_column(row, i, config.payload_format)?);
+ }
+ if name == config.tracking_column {
+ max_offset = value_as_string(&extract_column_value(row, i)?);
+ }
+ if name == config.pk_column {
+ row_pk = value_as_string(&extract_column_value(row, i)?);
+ }
+ }
+ }
+
+ if extracted_payload.is_none() {
+ let mut data = serde_json::Map::new();
+ for (i, column) in row.columns().iter().enumerate() {
+ let name = column.name();
+ let column_name = if config.snake_case_columns {
+ to_snake_case(name)
+ } else {
+ name.to_string()
+ };
+ let value = extract_column_value(row, i)?;
+ if name == config.tracking_column {
+ max_offset = value_as_string(&value);
+ }
+ if name == config.pk_column {
+ row_pk = value_as_string(&value);
+ }
+ data.insert(column_name, value);
+ }
+
+ extracted_payload = Some(if config.include_metadata {
+ let record = DatabaseRecord {
+ table_name: config.table.to_string(),
+ operation_type: "SELECT".to_string(),
+ timestamp: Utc::now(),
+ data: serde_json::Value::Object(data),
+ old_data: None,
+ };
+ simd_json::to_vec(&record).map_err(|_| Error::InvalidRecord)?
+ } else {
+ simd_json::to_vec(&data).map_err(|_| Error::InvalidRecord)?
+ });
+ }
+
+ // Both paths above always assign extracted_payload before reaching here.
+ Ok(build_processed_row(
+ extracted_payload.unwrap(),
+ max_offset,
+ row_pk,
+ ))
+ }
+}
+
+fn extract_column_value(row: &MySqlRow, column_index: usize) -> Result<serde_json::Value, Error> {
+ let column = &row.columns()[column_index];
+ let type_name = column.type_info().name();
+
+ match type_name {
+ "BOOLEAN" => {
+ let value: Option<bool> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::Bool)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT" => {
+ let value: Option<i8> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT" => {
+ let value: Option<i16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT" | "INT" => {
+ let value: Option<i32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as i64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT" => {
+ let value: Option<i64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TINYINT UNSIGNED" => {
+ let value: Option<u8> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "SMALLINT UNSIGNED" => {
+ let value: Option<u16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "MEDIUMINT UNSIGNED" | "INT UNSIGNED" => {
+ let value: Option<u32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIGINT UNSIGNED" => {
+ let value: Option<u64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "FLOAT" => {
+ let value: Option<f32> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as f64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DOUBLE" => {
+ let value: Option<f64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DECIMAL" => {
+ let value: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .and_then(|s| s.parse::<f64>().ok())
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BIT" => {
+ let value: Option<u64> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::from)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "YEAR" => {
+ let value: Option<u16> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|v| serde_json::Value::from(v as u64))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATE" => {
+ let value: Option<NaiveDate> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|d| serde_json::Value::String(d.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "TIME" => {
+ let value: Option<NaiveTime> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|t| serde_json::Value::String(t.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "DATETIME" | "TIMESTAMP" => {
+ let value: Option<NaiveDateTime> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|dt| serde_json::Value::String(dt.to_string()))
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT" | "MEDIUMTEXT" | "LONGTEXT" | "ENUM" | "SET" => {
+ let value: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "BINARY" | "VARBINARY" | "TINYBLOB" | "BLOB" | "MEDIUMBLOB" | "LONGBLOB" | "GEOMETRY" => {
+ let value: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value
+ .map(|bytes| {
+ serde_json::Value::String(
+ base64::engine::general_purpose::STANDARD.encode(&bytes),
+ )
+ })
+ .unwrap_or(serde_json::Value::Null))
+ }
+ "JSON" => {
+ let value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(value.unwrap_or(serde_json::Value::Null))
+ }
+ "NULL" => Ok(serde_json::Value::Null),
+ _ => {
+ let column_name = column.name();
+ warn!(
+ "Column '{column_name}' has unrecognized MySQL type '{type_name}', \
+ attempting text extraction"
+ );
+ if let Ok(text) = row.try_get::<Option<String>, _>(column_index) {
+ return Ok(text
+ .map(serde_json::Value::String)
+ .unwrap_or(serde_json::Value::Null));
+ }
+ if let Ok(bytes) = row.try_get::<Option<Vec<u8>>, _>(column_index) {
+ return Ok(bytes
+ .map(|b| {
+ serde_json::Value::String(
+ base64::engine::general_purpose::STANDARD.encode(&b),
+ )
+ })
+ .unwrap_or(serde_json::Value::Null));
+ }
+ error!(
+ "Column '{column_name}' has unsupported MySQL type '{type_name}', \
+ returning null"
+ );
+ Ok(serde_json::Value::Null)
+ }
+ }
+}
+
+fn value_as_string(value: &serde_json::Value) -> Option<String> {
+ match value {
+ serde_json::Value::String(s) => Some(s.clone()),
+ serde_json::Value::Number(n) => Some(n.to_string()),
+ _ => None,
+ }
+}
+
+fn build_processed_row(
+ payload: Vec<u8>,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+) -> ProcessedRow {
+ let now = Utc::now().timestamp_millis() as u64;
+ ProcessedRow {
+ message: ProducedMessage {
+ id: Some(Uuid::new_v4().as_u128()),
+ headers: None,
+ checksum: None,
+ timestamp: Some(now),
+ origin_timestamp: Some(now),
+ payload,
+ },
+ max_offset,
+ row_pk,
+ }
+}
+
+fn to_snake_case(input: &str) -> String {
+ let mut result = String::new();
+ let mut prev_was_uppercase = false;
+ for (i, ch) in input.chars().enumerate() {
+ if ch.is_uppercase() {
+ if i > 0 && !prev_was_uppercase {
+ result.push('_');
+ }
+ if let Some(lc) = ch.to_lowercase().next() {
+ result.push(lc);
+ } else {
+ result.push(ch);
+ }
+ prev_was_uppercase = true;
+ } else {
+ result.push(ch);
+ prev_was_uppercase = false;
+ }
+ }
+ result
+}
+
+fn redact_connection_string(conn_str: &str) -> String {
+ if let Some(scheme_end) = conn_str.find("://") {
+ let scheme = &conn_str[..scheme_end + 3];
+ let rest = &conn_str[scheme_end + 3..];
+ let preview: String = rest.chars().take(3).collect();
+ return format!("{scheme}{preview}***");
+ }
+ let preview: String = conn_str.chars().take(3).collect();
+ format!("{preview}***")
+}
+
+fn quote_identifier(name: &str) -> Result<String, Error> {
+ if name.is_empty() {
+ return Err(Error::InvalidConfigValue(
+ "identifier must not be empty".to_string(),
+ ));
+ }
+ if name.contains('\0') {
+ return Err(Error::InvalidConfigValue(format!(
+ "identifier '{name}' contains NUL byte"
+ )));
+ }
+ let escaped = name.replace('`', "``");
+ Ok(format!("`{escaped}`"))
+}
+
+fn quote_qualified_identifier(name: &str) -> Result<String, Error> {
+ if !name.contains('.') {
+ return quote_identifier(name);
+ }
+ let parts: Result<Vec<_>, _> = name.split('.').map(quote_identifier).collect();
+ Ok(parts?.join("."))
+}
+
+fn format_offset_value(value: &str) -> String {
+ if value.parse::<i64>().is_ok() || value.parse::<f64>().is_ok() {
Review Comment:
`"NaN".parse::<f64>()` and `"inf".parse::<f64>()` both return Ok in Rust, so this check passes for them and they get emitted bare into the WHERE clause. MySQL doesn't recognize `NaN`/`inf` as literals so the query fails at runtime. I think you want an additional `.is_ok_and(|v| v.is_finite())` guard on the f64 parse.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1358 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ if self.config.delete_after_read.unwrap_or(false) {
+ let delete_query =
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})");
+
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+
+ sqlx::query(sqlx::AssertSqlSafe(delete_query))
+ .execute(pool)
+ .await
+ .map_err(|e| {
+ error!("Failed to delete processed rows: {e}");
+ Error::InvalidRecord
+ })?;
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ let update_query = format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ );
+
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+
+ sqlx::query(sqlx::AssertSqlSafe(update_query))
+ .execute(pool)
+ .await
+ .map_err(|e| {
+ error!("Failed to mark rows as processed: {e}");
+ Error::InvalidRecord
+ })?;
+ }
+
+ Ok(())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+ let mut messages = Vec::new();
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Collect state updates to apply after processing
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for table in &self.config.tables {
+ let table_config = RowProcessingConfig {
+ table,
+ ..row_config
+ };
+
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.validate_custom_query(custom_query)?;
+ self.substitute_query_params(custom_query, table, &last_offset, batch_size)
+ } else {
+ self.build_polling_query(table, tracking_column, &last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut max_offset: Option<String> = None;
+ let mut processed_ids: Vec<String> = Vec::new();
+
+ let mut count_per_table = 0;
+ for row in rows {
+ let processed = self.process_row(&row, &table_config)?;
+
+ if let Some(pk) = processed.row_pk {
+ processed_ids.push(pk);
+ }
+ if let Some(offset) = processed.max_offset {
+ max_offset = Some(offset);
+ }
+
+ messages.push(processed.message);
+ count_per_table += 1;
+ total_processed += 1;
+ }
+
+ // Database I/O without holding the lock
+ if !processed_ids.is_empty() {
+ self.mark_or_delete_processed_rows(pool, table, pk_column, &processed_ids)
+ .await?;
+ }
Review Comment:
`mark_or_delete_processed_rows` runs inside `poll_tables()` before the messages reach the runtime's `producer.send()`. If the process crashes or publish fails after this point, those rows are gone from MySQL but never delivered to Iggy. It would be better to return the pending IDs in ProducedMessages and let the runtime execute the delete after confirmed send.
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,1358 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ max_offset: Option<String>,
+ row_pk: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") | Some("jsonb_direct") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+struct State {
+ last_poll_time: DateTime<Utc>,
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+ self.connect().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let poll_interval = self.poll_interval;
+ tokio::time::sleep(poll_interval).await;
+
+ let messages = self.poll_tables().await?;
+
+ let state = self.state.lock().await;
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ messages.len(),
+ state.processed_rows
+ );
+ }
+
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ let persisted_state = self.serialize_state(&state);
+
+ Ok(ProducedMessages {
+ schema,
+ messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(restored_state.unwrap_or(State {
+ last_poll_time: Utc::now(),
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ })),
+ verbose,
+ retry_delay,
+ poll_interval,
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> = row
+ .try_get(column_index)
+ .map_err(|_| Error::InvalidRecord)?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null))
+ .map_err(|_| Error::InvalidRecord)
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> String {
+ let offset_value = last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ .unwrap_or_default();
+
+ let now = Utc::now();
+
+ query
+ .replace("$table", table)
+ .replace("$offset", &offset_value)
+ .replace("$limit", &batch_size.to_string())
+ .replace("$now", &now.to_rfc3339())
+ .replace("$now_unix", &now.timestamp().to_string())
+ }
+
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let query_upper = query.to_uppercase();
+ if !query_upper.contains("SELECT") {
+ warn!("Custom query should contain SELECT statement");
+ }
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InvalidConfig);
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = last_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(offset)
+ ));
+ } else if let Some(initial) = &self.config.initial_offset {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(initial)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ pk_column: &str,
+ ids: &[String],
+ ) -> Result<(), Error> {
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(pk_column)?;
+
+ let ids_list = ids
+ .iter()
+ .map(|id| {
+ if id.parse::<i64>().is_ok() {
+ id.clone()
+ } else {
+ format!(
+ "'{}'",
+ id.replace('\\', "\\\\")
+ .replace('\'', "''")
+ .replace('\0', "")
+ )
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(", ");
+
+ if self.config.delete_after_read.unwrap_or(false) {
+ let delete_query =
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({ids_list})");
+
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+
+ sqlx::query(sqlx::AssertSqlSafe(delete_query))
+ .execute(pool)
+ .await
+ .map_err(|e| {
+ error!("Failed to delete processed rows: {e}");
+ Error::InvalidRecord
+ })?;
+ } else if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ let update_query = format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({ids_list})"
+ );
+
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+
+ sqlx::query(sqlx::AssertSqlSafe(update_query))
+ .execute(pool)
+ .await
+ .map_err(|e| {
+ error!("Failed to mark rows as processed: {e}");
+ Error::InvalidRecord
+ })?;
+ }
+
+ Ok(())
+ }
+
+ async fn poll_tables(&self) -> Result<Vec<ProducedMessage>, Error> {
+ let pool = self.get_pool()?;
+ let mut messages = Vec::new();
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.config.tracking_column.as_deref().unwrap_or("id");
+ let pk_column = self
+ .config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(tracking_column);
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Collect state updates to apply after processing
+ let mut state_updates: Vec<(String, String)> = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for table in &self.config.tables {
+ let table_config = RowProcessingConfig {
+ table,
+ ..row_config
+ };
+
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.validate_custom_query(custom_query)?;
+ self.substitute_query_params(custom_query, table, &last_offset, batch_size)
+ } else {
+ self.build_polling_query(table, tracking_column, &last_offset, batch_size)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await?;
+
+ let mut max_offset: Option<String> = None;
+ let mut processed_ids: Vec<String> = Vec::new();
+
+ let mut count_per_table = 0;
+ for row in rows {
+ let processed = self.process_row(&row, &table_config)?;
+
+ if let Some(pk) = processed.row_pk {
+ processed_ids.push(pk);
+ }
+ if let Some(offset) = processed.max_offset {
+ max_offset = Some(offset);
+ }
+
+ messages.push(processed.message);
+ count_per_table += 1;
+ total_processed += 1;
+ }
+
+ // Database I/O without holding the lock
+ if !processed_ids.is_empty() {
+ self.mark_or_delete_processed_rows(pool, table, pk_column, &processed_ids)
+ .await?;
Review Comment:
If `mark_or_delete_processed_rows` succeeds for table A but a later operation fails for table B, the `?` discards ALL collected messages (including A's). Table A's rows are already deleted/marked but its messages never get published. each table's batch needs independent error handling so one table's failure doesn't poison the rest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r4187509060
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,3317 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MySqlSourceConfig {
Review Comment:
Fixed, added #[serde(deny_unknown_fields)], so a misspelled key fails init. a typo can't enable DELETE (it defaults to false), the actual risk was the reverse, a requested delete/mark or other setting being silently ignored,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on PR #3568:
URL: https://github.com/apache/iggy/pull/3568#issuecomment-6002721522
/ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r4297516046
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,4071 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source,
+ source::SourceBatchResult, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+/// Upper bound on applying a batch's staged mark/delete after its Ack. The SDK waits
+/// `BATCH_RESULT_TIMEOUT` (30s) for the result, so this has to finish well inside it.
+const ACK_BATCH_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// MySQL rejects a prepared statement with more placeholders than this (error 1390), and
+/// the mark/delete binds one per row of the batch.
+const MAX_CLEANUP_BATCH_SIZE: u32 = 65_535;
+
+/// How many mark/delete statements in a row may fail before the connector stops. The
+/// operations stay in the persisted state and run again on restart.
+const MAX_CONSECUTIVE_CLEANUP_FAILURES: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ pending_batch: Mutex<Option<PendingBatch>>,
+ consecutive_cleanup_failures: AtomicU32,
+ verified_cleanup_keys: Mutex<HashSet<String>>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some("it has too few distinct values to be unique per row");
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some("binary values are compared as base64 text, which does not keep byte order");
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some("MySQL orders these by declaration index, not by label");
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL \
+ or not an ordered scalar (tinyint(1), JSON)"
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "row '{value}' is below the stored offset '{previous}' for tracking_column \
+ '{column}'. Filter the query with `{column} > $offset`"
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. Order the query ascending by '{column}'"
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+ /// Mark/delete work for a batch the runtime has already sent. It is persisted with
+ /// the batch's offsets so a crash between the send and the cleanup reruns it.
+ #[serde(default)]
+ pending_operations: Vec<PendingOperation>,
+}
+
+/// The rows of one table to mark or delete once their batch is acknowledged.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct PendingOperation {
+ table: String,
+ ids: Vec<PkValue>,
+ target: CleanupTarget,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct CleanupTarget {
+ key_column: String,
+ action: CleanupAction,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum CleanupAction {
+ Delete,
+ MarkProcessed { column: String },
+}
+
+/// The candidate state of the batch most recently handed to the runtime. Live state
+/// only takes it over on Ack, so a Nack leaves offsets and rows where they were.
+#[derive(Debug, Clone)]
+struct PendingBatch {
+ state: State,
+ /// The runtime has delivered the batch but its cleanup has not finished yet.
+ acknowledged: bool,
+ /// Cleanup finished and the next poll must persist `state` without the operations,
+ /// so a restart does not run them again.
+ retiring: bool,
+}
+
+struct PolledBatch {
+ messages: Vec<ProducedMessage>,
+ pending: Option<PendingBatch>,
+}
+
+enum CleanupKeyCheck {
+ Unique,
+ NotUnique,
+ TableMissing,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty, list the table even when custom_query hardcodes it"
+ .to_string(),
+ ));
+ }
+
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ if let Some(batch_size) = self.config.batch_size
+ && batch_size > MAX_CLEANUP_BATCH_SIZE
+ && self.cleanup_target().is_some()
+ {
+ return Err(Error::InitError(format!(
+ "batch_size {batch_size} exceeds {MAX_CLEANUP_BATCH_SIZE} with delete_after_read or \
+ processed_column set, MySQL allows at most {MAX_CLEANUP_BATCH_SIZE} placeholders \
+ in the statement that marks or deletes a batch"
+ )));
+ }
+
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "name the column custom_query orders by and filters with $offset"
+ } else {
+ "use a unique, monotonically increasing column"
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set, {guidance}"
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.validate_pending_cleanup()?;
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+ self.validate_cleanup_keys().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ if let Some(state) = self.retiring_state().await {
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: Some(self.serialize_pending_state(&state)?),
+ });
+ }
+
+ // Skip the pacing delay while draining a backlog, but never while a cleanup
+ // retry is pending, or a failing mark/delete would spin without a pause.
+ let cleanup_pending = self.pending_batch.lock().await.is_some();
+ if cleanup_pending || !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ {
+ let pending = self.pending_batch.lock().await;
+ if let Some(pending) = pending.as_ref() {
+ if !pending.acknowledged {
+ error!(
+ "{CONNECTOR_NAME} connector ID: {} was polled while a batch was still in flight",
+ self.id
+ );
+ return Err(Error::InvalidState);
+ }
+ // An empty batch whose Ack retries the cleanup the last one left behind.
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: None,
+ });
+ }
+ }
+
+ let polled = self.poll_tables().await?;
+
+ let processed_rows = match &polled.pending {
+ Some(pending) => pending.state.processed_rows,
+ None => self.state.lock().await.processed_rows,
+ };
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ }
+
+ // Idle cycles stage nothing, so return None to let the runtime skip the fsync.
+ let persisted_state = polled
+ .pending
+ .as_ref()
+ .map(|pending| self.serialize_pending_state(&pending.state))
+ .transpose()?;
+ *self.pending_batch.lock().await = polled.pending;
+
+ Ok(ProducedMessages {
+ schema,
+ messages: polled.messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> {
+ if result == SourceBatchResult::Nack {
+ // Dropping the candidate leaves the live offsets where they were, so the
+ // next poll reads the same rows again. Delivered batches keep their cleanup.
+ let mut pending = self.pending_batch.lock().await;
+ if pending
+ .as_ref()
+ .is_some_and(|pending| !pending.acknowledged && !pending.retiring)
+ {
+ pending.take();
+ }
+ return Ok(());
+ }
+
+ let Some(PendingBatch { mut state, .. }) = self.pending_batch.lock().await.clone() else {
+ return Ok(());
+ };
+
+ let operations = std::mem::take(&mut state.pending_operations);
+ let had_operations = !operations.is_empty();
+ let deadline = tokio::time::Instant::now() + ACK_BATCH_TIMEOUT;
+ let failed_operations = self.apply_pending_operations(operations, deadline).await?;
+
+ if failed_operations.is_empty() {
+ *self.state.lock().await = state.clone();
+ let mut pending = self.pending_batch.lock().await;
+ if had_operations {
+ *pending = Some(PendingBatch {
+ state,
+ acknowledged: false,
+ retiring: true,
+ });
+ } else {
+ pending.take();
+ }
+ } else {
+ state.pending_operations = failed_operations;
+ *self.pending_batch.lock().await = Some(PendingBatch {
+ state,
+ acknowledged: true,
+ retiring: false,
+ });
+ }
+
+ Ok(())
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ let state = restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ pending_operations: Vec::new(),
+ });
+ // The runtime only persists a batch's operations after sending it, so restored
+ // operations belong to a delivered batch and only the cleanup is left to do.
+ let pending_batch = (!state.pending_operations.is_empty()).then(|| PendingBatch {
+ state: state.clone(),
+ acknowledged: true,
+ retiring: false,
+ });
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(state),
+ pending_batch: Mutex::new(pending_batch),
+ consecutive_cleanup_failures: AtomicU32::new(0),
+ verified_cleanup_keys: Mutex::new(HashSet::new()),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ poisoned_tables: Mutex::new(HashSet::new()),
+ tracking_kinds: Mutex::new(HashMap::new()),
+ polls: AtomicU64::new(0),
+ cursor_regressions: Mutex::new(HashMap::new()),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ // `format_offset_value` escapes backslashes, which under NO_BACKSLASH_ESCAPES
+ // would reach MySQL as literal characters and filter on the wrong cursor.
+ .after_connect(|connection, _| {
+ Box::pin(async move {
+ sqlx::query(
+ "SET SESSION sql_mode = REPLACE(@@SESSION.sql_mode, 'NO_BACKSLASH_ESCAPES', '')",
+ )
+ .execute(&mut *connection)
+ .await?;
+ Ok(())
+ })
+ })
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn tracking_column(&self) -> &str {
+ self.config.tracking_column.as_deref().unwrap_or("id")
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn resolve_tracking_columns(&self) -> Result<HashMap<String, OffsetKind>, Error> {
+ // Whether the connector owns the query, and can therefore hold the schema to
+ // what that query will do with it.
+ let strict = self.config.custom_query.is_none();
+ let pool = self.get_pool()?;
+ let tracking_column = self.tracking_column();
+ let mut kinds = HashMap::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ }
+
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ }
+
+ warn!(
+ "tracking_column '{tracking_column}' is not a column of table '{table}'; \
+ assuming custom_query projects it. Its comparison order is resolved from \
+ the first result set instead."
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ // The column's type survives a plain projection, so it is worth rejecting
+ // whatever the query looks like. A custom_query aliasing an expression onto
+ // a name the table also uses is the one false positive, hence the hint.
+ if let Some(reason) = unusable_tracking_type(&data_type) {
+ let aliasing_hint = if self.config.custom_query.is_some() {
+ " If custom_query aliases a different expression to this name, rename the \
+ alias so it does not collide with the column."
+ } else {
+ ""
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': {reason}. Use an integer, timestamp, or string \
+ column.{aliasing_hint}"
+ )));
+ }
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; a row \
+ whose value is NULL cannot produce a cursor and will fail that table's batch. \
+ Have custom_query exclude those rows, or declare the column NOT NULL."
+ );
+ }
+
+ if is_collation_ordered(&data_type) {
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': values must be unique and increase in its collation order \
+ (zero-pad numbers, prefer a _bin collation)"
+ );
+ }
+
+ kinds.insert(table.clone(), offset_kind_for_data_type(&data_type));
+ }
+ Ok(kinds)
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn serialize_pending_state(&self, state: &State) -> Result<ConnectorState, Error> {
+ self.serialize_state(state).ok_or_else(|| {
+ Error::Serialization(format!("failed to serialize {CONNECTOR_NAME} state"))
+ })
+ }
+
+ async fn retiring_state(&self) -> Option<State> {
+ self.pending_batch
+ .lock()
+ .await
+ .as_ref()
+ .filter(|pending| pending.retiring)
+ .map(|pending| pending.state.clone())
+ }
+
+ fn primary_key_column(&self) -> &str {
+ self.config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(self.tracking_column())
+ }
+
+ fn cleanup_target(&self) -> Option<CleanupTarget> {
+ let action = if self.config.delete_after_read.unwrap_or(false) {
+ CleanupAction::Delete
+ } else if let Some(column) = &self.config.processed_column {
+ CleanupAction::MarkProcessed {
+ column: column.clone(),
+ }
+ } else {
+ return None;
+ };
+ Some(CleanupTarget {
+ key_column: self.primary_key_column().to_string(),
+ action,
+ })
+ }
+
+ /// Restored operations were staged under the config of the run that wrote them.
+ /// Running them under a different key or action would touch rows nobody read.
+ fn validate_pending_cleanup(&mut self) -> Result<(), Error> {
+ let configured = self.cleanup_target();
+ for operation in &self.state.get_mut().pending_operations {
+ if configured.as_ref() != Some(&operation.target) {
+ return Err(Error::InitError(format!(
+ "the cleanup configuration changed while cleanup of {} delivered row(s) in \
+ table '{}' is pending; restore primary_key_column '{}' and action {:?}",
+ operation.ids.len(),
+ operation.table,
+ operation.target.key_column,
+ operation.target.action
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ /// `WHERE key IN (...)` on a key that repeats marks or deletes every row sharing a
+ /// value, including ones that were never read. Tables that don't exist yet are
+ /// checked by the first poll that stages cleanup for them.
+ async fn validate_cleanup_keys(&mut self) -> Result<(), Error> {
+ let Some(target) = self.cleanup_target() else {
+ return Ok(());
+ };
+ let pool = self.get_pool()?;
+ let mut verified = HashSet::with_capacity(self.config.tables.len());
+ for table in &self.config.tables {
+ let check = cleanup_key_check(pool, table, &target.key_column)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to check primary_key_column '{}' on table '{table}': {e}",
+ target.key_column
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ verified.insert(table.clone());
+ }
+ CleanupKeyCheck::NotUnique => {
+ return Err(Error::InitError(not_unique_key_message(
+ table,
+ &target.key_column,
+ )));
+ }
+ CleanupKeyCheck::TableMissing => {}
+ }
+ }
+ *self.verified_cleanup_keys.get_mut() = verified;
+ Ok(())
+ }
+
+ async fn cleanup_key_is_unique(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ key_column: &str,
+ ) -> Result<bool, Error> {
+ if self.verified_cleanup_keys.lock().await.contains(table) {
+ return Ok(true);
+ }
+ let check = cleanup_key_check(pool, table, key_column)
+ .await
+ .map_err(|e| {
+ Error::Connection(format!(
+ "failed to check primary_key_column '{key_column}' on table '{table}': {e}"
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ self.verified_cleanup_keys
+ .lock()
+ .await
+ .insert(table.to_string());
+ Ok(true)
+ }
+ CleanupKeyCheck::NotUnique => Ok(false),
+ CleanupKeyCheck::TableMissing => Err(Error::Connection(format!(
+ "table '{table}' disappeared before its primary_key_column could be checked"
+ ))),
+ }
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ /// Best current knowledge of how a table's tracking values compare.
+ async fn tracking_kind(&self, table: &str) -> OffsetKind {
+ self.tracking_kinds
+ .lock()
+ .await
+ .get(table)
+ .copied()
+ .unwrap_or(OffsetKind::Unknown)
+ }
+
+ /// Whether the stored cursor is what limits the rows a poll can see. The built
+ /// polling query always filters on it; a `custom_query` does so only if it
+ /// interpolates `$offset`.
+ fn cursor_gates_query(&self) -> bool {
+ match &self.config.custom_query {
+ Some(query) => query.contains("$offset"),
+ None => true,
+ }
+ }
+
+ /// The offset a poll actually resumes from: the stored cursor, or the configured
+ /// starting point before anything has been stored.
+ fn effective_offset(&self, last_offset: &Option<String>) -> Option<String> {
+ last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let offset_value = self.effective_offset(last_offset).unwrap_or_default();
+ let offset = format_offset_value(&offset_value, kind);
+ let quoted_table = quote_qualified_identifier(table)?;
+ let limit = batch_size.to_string();
+ let now = Utc::now();
+ let now_unix = now.timestamp().to_string();
+ let now = now.to_rfc3339();
+ let replacements = [
+ ("$table", quoted_table.as_str()),
+ ("$offset", offset.as_str()),
+ ("$limit", limit.as_str()),
+ ("$now_unix", now_unix.as_str()),
+ ("$now", now.as_str()),
+ ];
+
+ Ok(substitute_query_tokens(query, &replacements))
+ }
+
+ /// The query is substituted as text and re-executed once per configured table on
+ /// every poll for the life of the process, so a shape that cannot terminate does
+ /// not fail, it repeats. What can be judged from the text alone is judged here,
+ /// at startup, rather than left to the first poll.
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let tracking_column = self.tracking_column();
+
+ match leading_statement_token(query) {
+ None => {
+ return Err(Error::InitError(
+ "custom_query has no statement: it is empty, or nothing but comments"
+ .to_string(),
+ ));
+ }
+ Some(token) if !READ_STATEMENT_STARTS.contains(&token.as_str()) => {
+ return Err(Error::InitError(format!(
+ "custom_query must be a read starting with SELECT, WITH or TABLE, found \
+ '{token}'"
+ )));
+ }
+ Some(_) => {}
+ }
+
+ // Collapsed so `ORDER\nBY` and `ORDER BY` read the same as `ORDER BY`; the
+ // check hard-fails, so a formatting difference must not decide it.
+ let normalized = query
+ .split_whitespace()
+ .collect::<Vec<_>>()
+ .join(" ")
+ .to_uppercase();
+
+ if query.contains("$offset") {
+ if !normalized.contains("ORDER BY") {
+ return Err(Error::InitError(format!(
+ "custom_query filters on $offset but has no ORDER BY, add \
+ `ORDER BY {tracking_column}`"
+ )));
+ }
+ } else if !self.config.delete_after_read.unwrap_or(false)
+ && self.config.processed_column.is_none()
+ {
+ return Err(Error::InitError(format!(
+ "custom_query must filter with $offset (`WHERE {tracking_column} > $offset`), or \
+ set delete_after_read or processed_column"
+ )));
+ }
+
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "custom_query uses $table but no tables are configured".to_string(),
+ ));
+ }
+ // Without $table the same SQL is executed once per table, so identical rows
+ // would be emitted repeatedly, each carrying a different table's metadata,
+ // message ID, and tracking offset. Table identity is not recoverable from a
+ // static query, so reject the combination instead of guessing one.
+ if !query.contains("$table") && self.config.tables.len() > 1 {
+ return Err(Error::InitError(format!(
+ "custom_query has no $table placeholder but {} tables are configured ({}), use \
+ $table in the query or configure a single table",
+ self.config.tables.len(),
+ self.config.tables.join(", ")
+ )));
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = self.effective_offset(last_offset) {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(&offset, kind)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn apply_pending_operations(
+ &self,
+ operations: Vec<PendingOperation>,
+ deadline: tokio::time::Instant,
+ ) -> Result<Vec<PendingOperation>, Error> {
+ let mut failed_operations = Vec::new();
+ let mut operations = operations.into_iter();
+ while let Some(operation) = operations.next() {
+ if tokio::time::Instant::now() >= deadline {
+ failed_operations.push(operation);
+ failed_operations.extend(operations);
+ break;
+ }
+ let result = tokio::time::timeout_at(deadline, async {
+ let pool = self.get_pool()?;
+ self.mark_or_delete_processed_rows(pool, &operation).await
+ })
+ .await;
+ match result {
+ Ok(Ok(())) => {
+ self.consecutive_cleanup_failures
+ .store(0, Ordering::Relaxed);
+ }
+ Ok(Err(error)) => {
+ self.record_cleanup_failure(&operation, &error)?;
+ failed_operations.push(operation);
+ }
+ Err(_) => {
+ self.record_cleanup_failure(
+ &operation,
+ "cleanup exceeded the 10s budget for one Ack",
+ )?;
+ failed_operations.push(operation);
+ }
+ }
+ }
+ Ok(failed_operations)
+ }
+
+ fn record_cleanup_failure(
+ &self,
+ operation: &PendingOperation,
+ error: impl std::fmt::Display,
+ ) -> Result<(), Error> {
+ let consecutive_failures = self
+ .consecutive_cleanup_failures
+ .fetch_add(1, Ordering::Relaxed)
+ .saturating_add(1);
+ error!(
+ "Failed to mark or delete {} delivered row(s) in table '{}' for {CONNECTOR_NAME} \
+ connector ID: {}, retrying on the next poll ({consecutive_failures} consecutive \
+ failures): {error}",
+ operation.ids.len(),
+ operation.table,
+ self.id
+ );
+ if consecutive_failures >= MAX_CONSECUTIVE_CLEANUP_FAILURES {
+ return Err(Error::Connection(format!(
+ "stopping {CONNECTOR_NAME} connector ID {} after {consecutive_failures} \
+ consecutive mark/delete failures; the cleanup stays in the saved state and \
+ runs again on restart",
+ self.id
+ )));
+ }
+ Ok(())
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ operation: &PendingOperation,
+ ) -> Result<(), Error> {
+ let PendingOperation { table, ids, target } = operation;
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(&target.key_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = match &target.action {
+ CleanupAction::Delete => {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
Review Comment:
fixed.
Instead of saving a tracking value for every row, each pending cleanup now saves the batch's highest tracking value, and the cleanup only touches rows at or below it: DELETE FROM t WHERE pk IN (...) AND tracking_column <= batch max
A replacement row always gets a newer tracking value, so the cleanup leaves it alone. This also avoids exact-equality matching on DATETIME and DECIMAL values, and the state stays small.
Since this needs the tracking column to exist in the table, a computed or aliased tracking column is now rejected when delete_after_read or processed_column is set. The README covers it.
Added integration tests that replay a pending cleanup after a key is reused, for both the built-in query and a custom_query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): add MySQL source polling connector [iggy]
Posted by "tusharagrahari (via GitHub)" <gi...@apache.org>.
tusharagrahari commented on code in PR #3568:
URL: https://github.com/apache/iggy/pull/3568#discussion_r4306754418
##########
core/connectors/sources/mysql_source/src/lib.rs:
##########
@@ -0,0 +1,4071 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use base64::Engine;
+use chrono::{NaiveDate, NaiveDateTime, NaiveTime};
+use humantime::Duration as HumanDuration;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source,
+ source::SourceBatchResult, source_connector,
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use sqlx::mysql::{MySqlDatabaseError, MySqlRow};
+use sqlx::{Column, MySql, Pool, Row, TypeInfo, mysql::MySqlPoolOptions};
+use std::cmp::Ordering as CmpOrdering;
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering};
+use std::time::Duration;
+use tokio::sync::Mutex;
+use tracing::{debug, error, info, warn};
+use uuid::Uuid;
+
+source_connector!(MySqlSource);
+
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+
+/// Fixed namespace so a given (table, key) always hashes to the same message id
+/// across restarts. This is what makes a replayed row idempotent for downstream
+/// dedup, so the value must never change once connectors are in the field.
+const MESSAGE_ID_NAMESPACE: Uuid = Uuid::from_u128(0x8f3b1e6a4c9d4f2a9b7c0d1e2f3a4b5c);
+
+/// A disabled table is skipped before anything is logged, so without a reminder the
+/// only evidence a connector went dark is the single line written when it happened.
+/// At the default 10s poll interval this repeats it every ten minutes.
+const DISABLED_TABLES_LOG_INTERVAL: u64 = 60;
+
+/// How many consecutive polls a table may end in `CursorRegression` before the repeat
+/// stops reading as transient. Nothing is published in the meantime, so past this it
+/// is a stall the operator has to see.
+const CURSOR_REGRESSION_ERROR_THRESHOLD: u32 = 3;
+
+/// Upper bound on applying a batch's staged mark/delete after its Ack. The SDK waits
+/// `BATCH_RESULT_TIMEOUT` (30s) for the result, so this has to finish well inside it.
+const ACK_BATCH_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// MySQL rejects a prepared statement with more placeholders than this (error 1390), and
+/// the mark/delete binds one per row of the batch.
+const MAX_CLEANUP_BATCH_SIZE: u32 = 65_535;
+
+/// How many mark/delete statements in a row may fail before the connector stops. The
+/// operations stay in the persisted state and run again on restart.
+const MAX_CONSECUTIVE_CLEANUP_FAILURES: u32 = 3;
+
+#[derive(Debug)]
+pub struct MySqlSource {
+ pub id: u32,
+ pool: Option<Pool<MySql>>,
+ config: MySqlSourceConfig,
+ state: Mutex<State>,
+ pending_batch: Mutex<Option<PendingBatch>>,
+ consecutive_cleanup_failures: AtomicU32,
+ verified_cleanup_keys: Mutex<HashSet<String>>,
+ verbose: bool,
+ retry_delay: Duration,
+ poll_interval: Duration,
+ last_batch_full: AtomicBool,
+ poisoned_tables: Mutex<HashSet<String>>,
+ tracking_kinds: Mutex<HashMap<String, OffsetKind>>,
+ polls: AtomicU64,
+ cursor_regressions: Mutex<HashMap<String, u32>>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct MySqlSourceConfig {
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")]
+ pub connection_string: SecretString,
+ pub tables: Vec<String>,
+ pub poll_interval: Option<String>,
+ pub batch_size: Option<u32>,
+ pub tracking_column: Option<String>,
+ pub initial_offset: Option<String>,
+ pub max_connections: Option<u32>,
+ pub custom_query: Option<String>,
+ pub snake_case_columns: Option<bool>,
+ pub include_metadata: Option<bool>,
+ pub delete_after_read: Option<bool>,
+ pub processed_column: Option<String>,
+ pub primary_key_column: Option<String>,
+ pub payload_column: Option<String>,
+ pub payload_format: Option<String>,
+ pub verbose_logging: Option<bool>,
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum PayloadFormat {
+ #[default]
+ Json,
+ Bytea,
+ Text,
+ JsonDirect,
+}
+
+struct ProcessedRow {
+ message: ProducedMessage,
+ tracking_value: Option<String>,
+ row_pk: Option<PkValue>,
+}
+
+/// Why a table's batch could not be produced.
+enum FetchError {
+ Ordering(String),
+ /// The batch opened below the cursor it was told to resume from, so the query's
+ /// filter and the cursor disagree about what "past the offset" means. Retried
+ /// rather than latched, because unlike a within-batch decrease it depends on the
+ /// rows as much as on the query: a `custom_query` whose WHERE admits rows the
+ /// cursor does not gate (an OR branch is the usual one) only fails while such a
+ /// row is there, so it can clear without a config change.
+ CursorRegression(String),
+ MissingTrackingColumn(String),
+ UnusableTrackingValue(String),
+ StaleOffsetKind(String),
+ Other(Error),
+}
+
+impl From<Error> for FetchError {
+ fn from(error: Error) -> Self {
+ FetchError::Other(error)
+ }
+}
+
+/// How MySQL ordered a tracking column, which offsets alone cannot reveal: they are
+/// carried as strings, so an `INT` 10 and a `VARCHAR` "10" are indistinguishable by
+/// content while the server orders them differently.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum OffsetKind {
+ Numeric,
+ Lexical,
+ Unknown,
+}
+
+/// Maps an `information_schema.columns.data_type` to the order MySQL sorts it in.
+/// `decimal` belongs with the numbers even though `extract_column_value` renders it
+/// as a string to preserve precision.
+fn offset_kind_for_data_type(data_type: &str) -> OffsetKind {
+ const NUMERIC: &[&str] = &[
+ "tinyint",
+ "smallint",
+ "mediumint",
+ "int",
+ "integer",
+ "bigint",
+ "decimal",
+ "numeric",
+ "float",
+ "double",
+ "real",
+ "bit",
+ "year",
+ ];
+ if NUMERIC
+ .iter()
+ .any(|numeric| data_type.eq_ignore_ascii_case(numeric))
+ {
+ OffsetKind::Numeric
+ } else {
+ OffsetKind::Lexical
+ }
+}
+
+/// Whether MySQL orders the column by its collation. The other lexically compared
+/// types are the temporal ones, which render fixed-width and so compare the same way
+/// as text no matter what the collation is.
+fn is_collation_ordered(data_type: &str) -> bool {
+ const TEXT: &[&str] = &[
+ "char",
+ "varchar",
+ "tinytext",
+ "text",
+ "mediumtext",
+ "longtext",
+ ];
+ TEXT.iter().any(|text| data_type.eq_ignore_ascii_case(text))
+}
+
+fn offset_kind_for_type_name(type_name: &str) -> OffsetKind {
+ match type_name {
+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" | "TINYINT UNSIGNED"
+ | "SMALLINT UNSIGNED" | "MEDIUMINT UNSIGNED" | "INT UNSIGNED" | "BIGINT UNSIGNED"
+ | "FLOAT" | "DOUBLE" | "DECIMAL" | "BIT" | "YEAR" => OffsetKind::Numeric,
+ "DATE" | "TIME" | "DATETIME" | "TIMESTAMP" | "CHAR" | "VARCHAR" | "TINYTEXT" | "TEXT"
+ | "MEDIUMTEXT" | "LONGTEXT" => OffsetKind::Lexical,
+ _ => OffsetKind::Unknown,
+ }
+}
+
+fn unusable_tracking_type(data_type: &str) -> Option<&'static str> {
+ const BINARY: &[&str] = &[
+ "binary",
+ "varbinary",
+ "tinyblob",
+ "blob",
+ "mediumblob",
+ "longblob",
+ ];
+
+ if data_type.eq_ignore_ascii_case("json") {
+ return Some("it yields no ordered scalar cursor");
+ }
+ // Judged on `data_type` rather than on the `tinyint(1)` display width that spells
+ // BOOLEAN, because that width is not reliably reported: MySQL 8.0.19 and later drop
+ // it for unsigned columns while MariaDB keeps it.
+ if data_type.eq_ignore_ascii_case("tinyint") {
+ return Some("it has too few distinct values to be unique per row");
+ }
+ if BINARY
+ .iter()
+ .any(|binary| data_type.eq_ignore_ascii_case(binary))
+ {
+ return Some("binary values are compared as base64 text, which does not keep byte order");
+ }
+ if data_type.eq_ignore_ascii_case("enum") || data_type.eq_ignore_ascii_case("set") {
+ return Some("MySQL orders these by declaration index, not by label");
+ }
+ None
+}
+
+/// What a `custom_query` may begin with. `TABLE` is MySQL 8.0.19's table statement,
+/// and `(` opens a parenthesized query such as `(SELECT ...) UNION (SELECT ...)`.
+const READ_STATEMENT_STARTS: &[&str] = &["SELECT", "WITH", "TABLE", "("];
+
+/// The token that decides which statement a query is, skipping leading comments in
+/// all three MySQL spellings. Returns a single character when the statement does not
+/// open with a word, so a parenthesized query is still recognisable.
+fn leading_statement_token(query: &str) -> Option<String> {
+ let mut rest = query.trim_start();
+ loop {
+ rest = if let Some(after) = rest.strip_prefix("/*") {
+ after.split_once("*/")?.1.trim_start()
+ } else if rest.starts_with("--") || rest.starts_with('#') {
+ rest.split_once('\n')?.1.trim_start()
+ } else {
+ break;
+ };
+ }
+
+ let word: String = rest
+ .chars()
+ .take_while(|c| c.is_ascii_alphanumeric() || *c == '_')
+ .collect();
+ if word.is_empty() {
+ return rest.chars().next().map(String::from);
+ }
+ Some(word.to_uppercase())
+}
+
+/// The order the tracking column sorts in according to the result set that just
+/// came back, or `None` when the query did not project the column under that name.
+fn observed_tracking_kind(row: &MySqlRow, tracking_column: &str) -> Option<OffsetKind> {
+ row.columns()
+ .iter()
+ .find(|column| column.name().eq_ignore_ascii_case(tracking_column))
+ .map(|column| offset_kind_for_type_name(column.type_info().name()))
+ .filter(|kind| *kind != OffsetKind::Unknown)
+}
+
+/// `Clone` so the passing case, which is every batch, walks the names without collecting.
+fn ensure_tracking_column_projected<'a>(
+ projected: impl IntoIterator<Item = &'a str> + Clone,
+ tracking_column: &str,
+) -> Result<(), FetchError> {
+ if projected
+ .clone()
+ .into_iter()
+ .any(|column| column.eq_ignore_ascii_case(tracking_column))
+ {
+ return Ok(());
+ }
+ Err(FetchError::MissingTrackingColumn(format!(
+ "the result set has no column '{tracking_column}', it returned [{}]",
+ projected.into_iter().collect::<Vec<_>>().join(", ")
+ )))
+}
+
+fn unusable_tracking_value(tracking_column: &str, pk: &str) -> FetchError {
+ FetchError::UnusableTrackingValue(format!(
+ "row (pk {pk}) has no usable value for tracking_column '{tracking_column}': it is NULL \
+ or not an ordered scalar (tinyint(1), JSON)"
+ ))
+}
+
+/// Verifies rows arrive non-decreasing in the tracking column, which is what makes
+/// "cursor = last row's value" correct. The built polling query orders ascending by
+/// construction, so this only ever fires on a `custom_query`, where the operator
+/// owns the `ORDER BY`.
+struct OrderingGuard<'a> {
+ tracking_column: &'a str,
+ kind: OffsetKind,
+ last: Option<String>,
+ /// True while `last` still holds the cursor carried in from the previous poll
+ /// rather than a value from this batch. Distinguishes the two violations, and
+ /// keeps an empty batch from re-persisting the offset it started with.
+ at_cursor: bool,
+}
+
+enum OrderingViolation {
+ WithinBatch(String),
+ BelowCursor(String),
+}
+
+impl<'a> OrderingGuard<'a> {
+ fn new(tracking_column: &'a str, kind: OffsetKind, cursor: Option<String>) -> Self {
+ OrderingGuard {
+ tracking_column,
+ kind,
+ at_cursor: cursor.is_some(),
+ last: cursor,
+ }
+ }
+
+ fn accept(&mut self, value: String) -> Result<(), OrderingViolation> {
+ if let Some(previous) = &self.last
+ && offset_decreased(previous, &value, self.kind)
+ {
+ let column = self.tracking_column;
+ return Err(if self.at_cursor {
+ OrderingViolation::BelowCursor(format!(
+ "row '{value}' is below the stored offset '{previous}' for tracking_column \
+ '{column}'. Filter the query with `{column} > $offset`"
+ ))
+ } else {
+ OrderingViolation::WithinBatch(format!(
+ "rows arrived out of order by tracking_column '{column}': '{previous}' \
+ preceded '{value}'. Order the query ascending by '{column}'"
+ ))
+ });
+ }
+ self.last = Some(value);
+ self.at_cursor = false;
+ Ok(())
+ }
+
+ /// `None` when no row was accepted, so an empty batch leaves the stored offset
+ /// untouched rather than rewriting the cursor it was seeded with.
+ fn into_cursor(self) -> Option<String> {
+ if self.at_cursor { None } else { self.last }
+ }
+}
+
+struct TableBatch {
+ table: String,
+ messages: Vec<ProducedMessage>,
+ processed_ids: Vec<PkValue>,
+ max_offset: Option<String>,
+}
+
+impl PayloadFormat {
+ fn from_config(s: Option<&str>) -> Self {
+ match s.map(|s| s.to_lowercase()).as_deref() {
+ Some("bytea") | Some("raw") => PayloadFormat::Bytea,
+ Some("text") => PayloadFormat::Text,
+ Some("json_direct") | Some("jsonb") => PayloadFormat::JsonDirect,
+ _ => PayloadFormat::Json,
+ }
+ }
+}
+
+/// A primary-key value in a form the mark/delete `WHERE pk IN (...)` can bind
+/// faithfully. `Text` covers ints/strings/dates/decimals (MySQL implicitly
+/// converts the bound string to the column type). `Bytes` carries raw binary
+/// (BINARY/VARBINARY/BLOB) so a UUID stored as bytes matches the row instead of
+/// being compared as its base64 text, which never matches.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum PkValue {
+ Text(String),
+ Bytes(Vec<u8>),
+}
+
+impl PkValue {
+ fn as_key(&self) -> String {
+ match self {
+ PkValue::Text(text) => text.clone(),
+ PkValue::Bytes(bytes) => base64::engine::general_purpose::STANDARD.encode(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct DatabaseRecord {
+ pub table_name: String,
+ pub operation_type: String,
+ pub timestamp: DateTime<Utc>,
+ pub data: serde_json::Value,
+ pub old_data: Option<serde_json::Value>,
+}
+#[derive(Clone, Copy)]
+struct RowProcessingConfig<'a> {
+ table: &'a str,
+ tracking_column: &'a str,
+ pk_column: &'a str,
+ payload_format: PayloadFormat,
+ payload_col: &'a str,
+ snake_case_columns: bool,
+ include_metadata: bool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct State {
+ tracking_offsets: HashMap<String, String>,
+ processed_rows: u64,
+ /// Mark/delete work for a batch the runtime has already sent. It is persisted with
+ /// the batch's offsets so a crash between the send and the cleanup reruns it.
+ #[serde(default)]
+ pending_operations: Vec<PendingOperation>,
+}
+
+/// The rows of one table to mark or delete once their batch is acknowledged.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct PendingOperation {
+ table: String,
+ ids: Vec<PkValue>,
+ target: CleanupTarget,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+struct CleanupTarget {
+ key_column: String,
+ action: CleanupAction,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+enum CleanupAction {
+ Delete,
+ MarkProcessed { column: String },
+}
+
+/// The candidate state of the batch most recently handed to the runtime. Live state
+/// only takes it over on Ack, so a Nack leaves offsets and rows where they were.
+#[derive(Debug, Clone)]
+struct PendingBatch {
+ state: State,
+ /// The runtime has delivered the batch but its cleanup has not finished yet.
+ acknowledged: bool,
+ /// Cleanup finished and the next poll must persist `state` without the operations,
+ /// so a restart does not run them again.
+ retiring: bool,
+}
+
+struct PolledBatch {
+ messages: Vec<ProducedMessage>,
+ pending: Option<PendingBatch>,
+}
+
+enum CleanupKeyCheck {
+ Unique,
+ NotUnique,
+ TableMissing,
+}
+
+const CONNECTOR_NAME: &str = "MySQL source";
+
+#[async_trait]
+impl Source for MySqlSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ info!(
+ "Opening MySQL source connector with ID: {}, Tables: {:?}",
+ self.id, self.config.tables
+ );
+
+ if self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "tables must not be empty, list the table even when custom_query hardcodes it"
+ .to_string(),
+ ));
+ }
+
+ if self.config.batch_size == Some(0) {
+ return Err(Error::InitError(
+ "batch_size must be greater than 0; omit it to use the default of 1000".to_string(),
+ ));
+ }
+
+ if let Some(batch_size) = self.config.batch_size
+ && batch_size > MAX_CLEANUP_BATCH_SIZE
+ && self.cleanup_target().is_some()
+ {
+ return Err(Error::InitError(format!(
+ "batch_size {batch_size} exceeds {MAX_CLEANUP_BATCH_SIZE} with delete_after_read or \
+ processed_column set, MySQL allows at most {MAX_CLEANUP_BATCH_SIZE} placeholders \
+ in the statement that marks or deletes a batch"
+ )));
+ }
+
+ if self.config.batch_size == Some(1)
+ && self
+ .config
+ .custom_query
+ .as_deref()
+ .is_some_and(|query| query.contains("$offset"))
+ {
+ return Err(Error::InitError(
+ "batch_size must be greater than 1 when custom_query filters on $offset"
+ .to_string(),
+ ));
+ }
+
+ if self
+ .config
+ .tracking_column
+ .as_deref()
+ .is_none_or(str::is_empty)
+ {
+ let guidance = if self.config.custom_query.is_some() {
+ "name the column custom_query orders by and filters with $offset"
+ } else {
+ "use a unique, monotonically increasing column"
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column must be set, {guidance}"
+ )));
+ }
+
+ if let Some(ref col) = self.config.payload_column
+ && !col.is_empty()
+ && PayloadFormat::from_config(self.config.payload_format.as_deref())
+ == PayloadFormat::Json
+ {
+ return Err(Error::InitError(
+ "payload_format must be 'bytea', 'text', or 'json_direct' when payload_column is set"
+ .to_string(),
+ ));
+ }
+
+ // custom_query never changes after startup, so validate it once here
+ if let Some(ref query) = self.config.custom_query {
+ self.validate_custom_query(query)?;
+
+ if query.contains("$offset") && self.config.initial_offset.is_none() {
+ let state = self.state.lock().await;
+ let missing: Vec<&str> = self
+ .config
+ .tables
+ .iter()
+ .filter(|t| !state.tracking_offsets.contains_key(*t))
+ .map(String::as_str)
+ .collect();
+ if !missing.is_empty() {
+ return Err(Error::InitError(format!(
+ "custom_query uses $offset but initial_offset is not set and no stored offset exists for table(s): {}",
+ missing.join(", ")
+ )));
+ }
+ }
+ }
+
+ if self.config.delete_after_read.unwrap_or(false) && self.config.processed_column.is_some()
+ {
+ warn!(
+ "both delete_after_read and processed_column are set; delete_after_read takes precedence, so rows are deleted and processed_column only acts as a poll filter"
+ );
+ }
+
+ self.validate_pending_cleanup()?;
+ self.connect().await?;
+ *self.tracking_kinds.get_mut() = self.resolve_tracking_columns().await?;
+ self.validate_cleanup_keys().await?;
+
+ info!(
+ "MySQL source connector with ID: {} opened successfully",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ let schema = match self.payload_format() {
+ PayloadFormat::Bytea => Schema::Raw,
+ PayloadFormat::Text => Schema::Text,
+ PayloadFormat::JsonDirect | PayloadFormat::Json => Schema::Json,
+ };
+
+ if let Some(state) = self.retiring_state().await {
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: Some(self.serialize_pending_state(&state)?),
+ });
+ }
+
+ // Skip the pacing delay while draining a backlog, but never while a cleanup
+ // retry is pending, or a failing mark/delete would spin without a pause.
+ let cleanup_pending = self.pending_batch.lock().await.is_some();
+ if cleanup_pending || !self.last_batch_full.load(Ordering::Relaxed) {
+ tokio::time::sleep(self.poll_interval).await;
+ }
+
+ {
+ let pending = self.pending_batch.lock().await;
+ if let Some(pending) = pending.as_ref() {
+ if !pending.acknowledged {
+ error!(
+ "{CONNECTOR_NAME} connector ID: {} was polled while a batch was still in flight",
+ self.id
+ );
+ return Err(Error::InvalidState);
+ }
+ // An empty batch whose Ack retries the cleanup the last one left behind.
+ return Ok(ProducedMessages {
+ schema,
+ messages: Vec::new(),
+ state: None,
+ });
+ }
+ }
+
+ let polled = self.poll_tables().await?;
+
+ let processed_rows = match &polled.pending {
+ Some(pending) => pending.state.processed_rows,
+ None => self.state.lock().await.processed_rows,
+ };
+ if self.verbose {
+ info!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ } else {
+ debug!(
+ "MySQL source connector ID: {} produced {} messages. Total processed: {}",
+ self.id,
+ polled.messages.len(),
+ processed_rows
+ );
+ }
+
+ // Idle cycles stage nothing, so return None to let the runtime skip the fsync.
+ let persisted_state = polled
+ .pending
+ .as_ref()
+ .map(|pending| self.serialize_pending_state(&pending.state))
+ .transpose()?;
+ *self.pending_batch.lock().await = polled.pending;
+
+ Ok(ProducedMessages {
+ schema,
+ messages: polled.messages,
+ state: persisted_state,
+ })
+ }
+
+ async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> {
+ if result == SourceBatchResult::Nack {
+ // Dropping the candidate leaves the live offsets where they were, so the
+ // next poll reads the same rows again. Delivered batches keep their cleanup.
+ let mut pending = self.pending_batch.lock().await;
+ if pending
+ .as_ref()
+ .is_some_and(|pending| !pending.acknowledged && !pending.retiring)
+ {
+ pending.take();
+ }
+ return Ok(());
+ }
+
+ let Some(PendingBatch { mut state, .. }) = self.pending_batch.lock().await.clone() else {
+ return Ok(());
+ };
+
+ let operations = std::mem::take(&mut state.pending_operations);
+ let had_operations = !operations.is_empty();
+ let deadline = tokio::time::Instant::now() + ACK_BATCH_TIMEOUT;
+ let failed_operations = self.apply_pending_operations(operations, deadline).await?;
+
+ if failed_operations.is_empty() {
+ *self.state.lock().await = state.clone();
+ let mut pending = self.pending_batch.lock().await;
+ if had_operations {
+ *pending = Some(PendingBatch {
+ state,
+ acknowledged: false,
+ retiring: true,
+ });
+ } else {
+ pending.take();
+ }
+ } else {
+ state.pending_operations = failed_operations;
+ *self.pending_batch.lock().await = Some(PendingBatch {
+ state,
+ acknowledged: true,
+ retiring: false,
+ });
+ }
+
+ Ok(())
+ }
+
+ async fn close(&mut self) -> Result<(), Error> {
+ if let Some(pool) = self.pool.take() {
+ pool.close().await;
+ info!("MySQL connection pool closed for connector ID: {}", self.id);
+ }
+
+ let state = self.state.lock().await;
+ info!(
+ "MySQL source connector ID: {} closed. Total rows processed: {}",
+ self.id, state.processed_rows
+ );
+ Ok(())
+ }
+}
+
+impl MySqlSource {
+ pub fn new(id: u32, config: MySqlSourceConfig, state: Option<ConnectorState>) -> Self {
+ let verbose = config.verbose_logging.unwrap_or(false);
+ let restored_state = state
+ .and_then(|s| s.deserialize::<State>(CONNECTOR_NAME, id))
+ .inspect(|s| {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Tracking offsets: {:?}, processed rows: {}",
+ s.tracking_offsets, s.processed_rows
+ );
+ });
+
+ let delay_str = config.retry_delay.as_deref().unwrap_or(DEFAULT_RETRY_DELAY);
+ let retry_delay = HumanDuration::from_str(delay_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(1));
+ let interval_str = config.poll_interval.as_deref().unwrap_or("10s");
+ let poll_interval = HumanDuration::from_str(interval_str)
+ .map(|duration| duration.into())
+ .unwrap_or_else(|_| Duration::from_secs(10));
+ let state = restored_state.unwrap_or(State {
+ tracking_offsets: HashMap::new(),
+ processed_rows: 0,
+ pending_operations: Vec::new(),
+ });
+ // The runtime only persists a batch's operations after sending it, so restored
+ // operations belong to a delivered batch and only the cleanup is left to do.
+ let pending_batch = (!state.pending_operations.is_empty()).then(|| PendingBatch {
+ state: state.clone(),
+ acknowledged: true,
+ retiring: false,
+ });
+ MySqlSource {
+ id,
+ pool: None,
+ config,
+ state: Mutex::new(state),
+ pending_batch: Mutex::new(pending_batch),
+ consecutive_cleanup_failures: AtomicU32::new(0),
+ verified_cleanup_keys: Mutex::new(HashSet::new()),
+ verbose,
+ retry_delay,
+ poll_interval,
+ last_batch_full: AtomicBool::new(false),
+ poisoned_tables: Mutex::new(HashSet::new()),
+ tracking_kinds: Mutex::new(HashMap::new()),
+ polls: AtomicU64::new(0),
+ cursor_regressions: Mutex::new(HashMap::new()),
+ }
+ }
+
+ async fn connect(&mut self) -> Result<(), Error> {
+ let max_connections = self.config.max_connections.unwrap_or(10);
+ let redacted = redact_connection_string(self.config.connection_string.expose_secret());
+
+ info!("Connecting to MySQL with max {max_connections} connections: {redacted}");
+
+ let pool = MySqlPoolOptions::new()
+ .max_connections(max_connections)
+ // `format_offset_value` escapes backslashes, which under NO_BACKSLASH_ESCAPES
+ // would reach MySQL as literal characters and filter on the wrong cursor.
+ .after_connect(|connection, _| {
+ Box::pin(async move {
+ sqlx::query(
+ "SET SESSION sql_mode = REPLACE(@@SESSION.sql_mode, 'NO_BACKSLASH_ESCAPES', '')",
+ )
+ .execute(&mut *connection)
+ .await?;
+ Ok(())
+ })
+ })
+ .connect(self.config.connection_string.expose_secret())
+ .await
+ .map_err(|e| Error::InitError(format!("Failed to connect to MySQL: {e}")))?;
+
+ sqlx::query("SELECT 1")
+ .execute(&pool)
+ .await
+ .map_err(|e| Error::InitError(format!("Database connectivity test failed: {e}")))?;
+
+ self.pool = Some(pool);
+ info!("Connected to MySQL database with {max_connections} max connections");
+ Ok(())
+ }
+
+ fn tracking_column(&self) -> &str {
+ self.config.tracking_column.as_deref().unwrap_or("id")
+ }
+
+ /// Fail fast if a tracking column can't yield a stable, ordered scalar
+ /// cursor. `value_as_string` drops NULL, tinyint(1) (decoded as BOOLEAN),
+ /// and JSON to `None`, which silently stalls offset advancement and re-reads
+ /// rows every cycle.
+ async fn resolve_tracking_columns(&self) -> Result<HashMap<String, OffsetKind>, Error> {
+ // Whether the connector owns the query, and can therefore hold the schema to
+ // what that query will do with it.
+ let strict = self.config.custom_query.is_none();
+ let pool = self.get_pool()?;
+ let tracking_column = self.tracking_column();
+ let mut kinds = HashMap::with_capacity(self.config.tables.len());
+
+ for table in &self.config.tables {
+ let (schema, table_name) = table
+ .split_once('.')
+ .map(|(schema, name)| (schema.to_string(), name.to_string()))
+ .unwrap_or_else(|| (String::new(), table.clone()));
+
+ let row = sqlx::query(
+ "SELECT is_nullable, data_type, column_type \
+ FROM information_schema.columns \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ? AND column_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .bind(tracking_column)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to probe schema for tracking_column '{tracking_column}' on table '{table}': {e}"
+ ))
+ })?;
+
+ let Some(row) = row else {
+ // A missing columns row means either the table doesn't exist yet
+ // or the column is absent. Only the latter is a misconfiguration:
+ // a table created after the connector starts is legitimate, so
+ // defer validation rather than permanently disabling the source.
+ let table_exists = sqlx::query(
+ "SELECT 1 FROM information_schema.tables \
+ WHERE table_schema = IF(? = '', DATABASE(), ?) \
+ AND table_name = ?",
+ )
+ .bind(&schema)
+ .bind(&schema)
+ .bind(&table_name)
+ .fetch_optional(pool)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!("failed to probe existence of table '{table}': {e}"))
+ })?
+ .is_some();
+
+ if !table_exists {
+ warn!(
+ "table '{table}' does not exist yet; deferring tracking_column \
+ '{tracking_column}' validation until the table is created"
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ }
+
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' not found on table '{table}'"
+ )));
+ }
+
+ warn!(
+ "tracking_column '{tracking_column}' is not a column of table '{table}'; \
+ assuming custom_query projects it. Its comparison order is resolved from \
+ the first result set instead."
+ );
+ kinds.insert(table.clone(), OffsetKind::Unknown);
+ continue;
+ };
+
+ // Read by position: MySQL returns information_schema column names
+ // uppercased (IS_NULLABLE, ...), so try_get by lowercase name misses.
+ let is_nullable: String = row.try_get(0).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read is_nullable for table '{table}': {e}"
+ ))
+ })?;
+ let data_type: String = row.try_get(1).map_err(|e| {
+ Error::InitError(format!("failed to read data_type for table '{table}': {e}"))
+ })?;
+ let column_type: String = row.try_get(2).map_err(|e| {
+ Error::InitError(format!(
+ "failed to read column_type for table '{table}': {e}"
+ ))
+ })?;
+
+ // The column's type survives a plain projection, so it is worth rejecting
+ // whatever the query looks like. A custom_query aliasing an expression onto
+ // a name the table also uses is the one false positive, hence the hint.
+ if let Some(reason) = unusable_tracking_type(&data_type) {
+ let aliasing_hint = if self.config.custom_query.is_some() {
+ " If custom_query aliases a different expression to this name, rename the \
+ alias so it does not collide with the column."
+ } else {
+ ""
+ };
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': {reason}. Use an integer, timestamp, or string \
+ column.{aliasing_hint}"
+ )));
+ }
+
+ if is_nullable.eq_ignore_ascii_case("YES") {
+ if strict {
+ return Err(Error::InitError(format!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; \
+ a NULL value stalls offset tracking. Declare it NOT NULL."
+ )));
+ }
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' is nullable; a row \
+ whose value is NULL cannot produce a cursor and will fail that table's batch. \
+ Have custom_query exclude those rows, or declare the column NOT NULL."
+ );
+ }
+
+ if is_collation_ordered(&data_type) {
+ warn!(
+ "tracking_column '{tracking_column}' on table '{table}' has type \
+ '{column_type}': values must be unique and increase in its collation order \
+ (zero-pad numbers, prefer a _bin collation)"
+ );
+ }
+
+ kinds.insert(table.clone(), offset_kind_for_data_type(&data_type));
+ }
+ Ok(kinds)
+ }
+
+ fn payload_format(&self) -> PayloadFormat {
+ if let Some(ref payload_col) = self.config.payload_column
+ && !payload_col.is_empty()
+ {
+ return PayloadFormat::from_config(self.config.payload_format.as_deref());
+ }
+ PayloadFormat::Json
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn serialize_pending_state(&self, state: &State) -> Result<ConnectorState, Error> {
+ self.serialize_state(state).ok_or_else(|| {
+ Error::Serialization(format!("failed to serialize {CONNECTOR_NAME} state"))
+ })
+ }
+
+ async fn retiring_state(&self) -> Option<State> {
+ self.pending_batch
+ .lock()
+ .await
+ .as_ref()
+ .filter(|pending| pending.retiring)
+ .map(|pending| pending.state.clone())
+ }
+
+ fn primary_key_column(&self) -> &str {
+ self.config
+ .primary_key_column
+ .as_deref()
+ .unwrap_or(self.tracking_column())
+ }
+
+ fn cleanup_target(&self) -> Option<CleanupTarget> {
+ let action = if self.config.delete_after_read.unwrap_or(false) {
+ CleanupAction::Delete
+ } else if let Some(column) = &self.config.processed_column {
+ CleanupAction::MarkProcessed {
+ column: column.clone(),
+ }
+ } else {
+ return None;
+ };
+ Some(CleanupTarget {
+ key_column: self.primary_key_column().to_string(),
+ action,
+ })
+ }
+
+ /// Restored operations were staged under the config of the run that wrote them.
+ /// Running them under a different key or action would touch rows nobody read.
+ fn validate_pending_cleanup(&mut self) -> Result<(), Error> {
+ let configured = self.cleanup_target();
+ for operation in &self.state.get_mut().pending_operations {
+ if configured.as_ref() != Some(&operation.target) {
+ return Err(Error::InitError(format!(
+ "the cleanup configuration changed while cleanup of {} delivered row(s) in \
+ table '{}' is pending; restore primary_key_column '{}' and action {:?}",
+ operation.ids.len(),
+ operation.table,
+ operation.target.key_column,
+ operation.target.action
+ )));
+ }
+ }
+ Ok(())
+ }
+
+ /// `WHERE key IN (...)` on a key that repeats marks or deletes every row sharing a
+ /// value, including ones that were never read. Tables that don't exist yet are
+ /// checked by the first poll that stages cleanup for them.
+ async fn validate_cleanup_keys(&mut self) -> Result<(), Error> {
+ let Some(target) = self.cleanup_target() else {
+ return Ok(());
+ };
+ let pool = self.get_pool()?;
+ let mut verified = HashSet::with_capacity(self.config.tables.len());
+ for table in &self.config.tables {
+ let check = cleanup_key_check(pool, table, &target.key_column)
+ .await
+ .map_err(|e| {
+ Error::InitError(format!(
+ "failed to check primary_key_column '{}' on table '{table}': {e}",
+ target.key_column
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ verified.insert(table.clone());
+ }
+ CleanupKeyCheck::NotUnique => {
+ return Err(Error::InitError(not_unique_key_message(
+ table,
+ &target.key_column,
+ )));
+ }
+ CleanupKeyCheck::TableMissing => {}
+ }
+ }
+ *self.verified_cleanup_keys.get_mut() = verified;
+ Ok(())
+ }
+
+ async fn cleanup_key_is_unique(
+ &self,
+ pool: &Pool<MySql>,
+ table: &str,
+ key_column: &str,
+ ) -> Result<bool, Error> {
+ if self.verified_cleanup_keys.lock().await.contains(table) {
+ return Ok(true);
+ }
+ let check = cleanup_key_check(pool, table, key_column)
+ .await
+ .map_err(|e| {
+ Error::Connection(format!(
+ "failed to check primary_key_column '{key_column}' on table '{table}': {e}"
+ ))
+ })?;
+ match check {
+ CleanupKeyCheck::Unique => {
+ self.verified_cleanup_keys
+ .lock()
+ .await
+ .insert(table.to_string());
+ Ok(true)
+ }
+ CleanupKeyCheck::NotUnique => Ok(false),
+ CleanupKeyCheck::TableMissing => Err(Error::Connection(format!(
+ "table '{table}' disappeared before its primary_key_column could be checked"
+ ))),
+ }
+ }
+
+ fn get_pool(&self) -> Result<&Pool<MySql>, Error> {
+ self.pool
+ .as_ref()
+ .ok_or_else(|| Error::InitError("Database not connected".to_string()))
+ }
+
+ fn extract_payload_column(
+ &self,
+ row: &MySqlRow,
+ column_index: usize,
+ format: PayloadFormat,
+ ) -> Result<Vec<u8>, Error> {
+ let column_name = row.columns()[column_index].name();
+ match format {
+ PayloadFormat::Bytea => {
+ let bytes: Option<Vec<u8>> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as bytea: {e}"
+ ))
+ })?;
+ Ok(bytes.unwrap_or_default())
+ }
+ PayloadFormat::Text => {
+ let text: Option<String> = row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as text (invalid UTF-8?): {e}"
+ ))
+ })?;
+ Ok(text.unwrap_or_default().into_bytes())
+ }
+ PayloadFormat::JsonDirect => {
+ let json_value: Option<serde_json::Value> =
+ row.try_get(column_index).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}' as json_direct (invalid JSON?): {e}"
+ ))
+ })?;
+ simd_json::to_vec(&json_value.unwrap_or(serde_json::Value::Null)).map_err(|e| {
+ Error::InvalidRecordValue(format!(
+ "payload column '{column_name}': failed to serialize JSON: {e}"
+ ))
+ })
+ }
+ PayloadFormat::Json => Err(Error::InvalidConfig), // unreachable! if payload_column is there then payload_format can never be json
+ }
+ }
+
+ /// Best current knowledge of how a table's tracking values compare.
+ async fn tracking_kind(&self, table: &str) -> OffsetKind {
+ self.tracking_kinds
+ .lock()
+ .await
+ .get(table)
+ .copied()
+ .unwrap_or(OffsetKind::Unknown)
+ }
+
+ /// Whether the stored cursor is what limits the rows a poll can see. The built
+ /// polling query always filters on it; a `custom_query` does so only if it
+ /// interpolates `$offset`.
+ fn cursor_gates_query(&self) -> bool {
+ match &self.config.custom_query {
+ Some(query) => query.contains("$offset"),
+ None => true,
+ }
+ }
+
+ /// The offset a poll actually resumes from: the stored cursor, or the configured
+ /// starting point before anything has been stored.
+ fn effective_offset(&self, last_offset: &Option<String>) -> Option<String> {
+ last_offset
+ .clone()
+ .or_else(|| self.config.initial_offset.clone())
+ }
+
+ fn substitute_query_params(
+ &self,
+ query: &str,
+ table: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let offset_value = self.effective_offset(last_offset).unwrap_or_default();
+ let offset = format_offset_value(&offset_value, kind);
+ let quoted_table = quote_qualified_identifier(table)?;
+ let limit = batch_size.to_string();
+ let now = Utc::now();
+ let now_unix = now.timestamp().to_string();
+ let now = now.to_rfc3339();
+ let replacements = [
+ ("$table", quoted_table.as_str()),
+ ("$offset", offset.as_str()),
+ ("$limit", limit.as_str()),
+ ("$now_unix", now_unix.as_str()),
+ ("$now", now.as_str()),
+ ];
+
+ Ok(substitute_query_tokens(query, &replacements))
+ }
+
+ /// The query is substituted as text and re-executed once per configured table on
+ /// every poll for the life of the process, so a shape that cannot terminate does
+ /// not fail, it repeats. What can be judged from the text alone is judged here,
+ /// at startup, rather than left to the first poll.
+ fn validate_custom_query(&self, query: &str) -> Result<(), Error> {
+ let tracking_column = self.tracking_column();
+
+ match leading_statement_token(query) {
+ None => {
+ return Err(Error::InitError(
+ "custom_query has no statement: it is empty, or nothing but comments"
+ .to_string(),
+ ));
+ }
+ Some(token) if !READ_STATEMENT_STARTS.contains(&token.as_str()) => {
+ return Err(Error::InitError(format!(
+ "custom_query must be a read starting with SELECT, WITH or TABLE, found \
+ '{token}'"
+ )));
+ }
+ Some(_) => {}
+ }
+
+ // Collapsed so `ORDER\nBY` and `ORDER BY` read the same as `ORDER BY`; the
+ // check hard-fails, so a formatting difference must not decide it.
+ let normalized = query
+ .split_whitespace()
+ .collect::<Vec<_>>()
+ .join(" ")
+ .to_uppercase();
+
+ if query.contains("$offset") {
+ if !normalized.contains("ORDER BY") {
+ return Err(Error::InitError(format!(
+ "custom_query filters on $offset but has no ORDER BY, add \
+ `ORDER BY {tracking_column}`"
+ )));
+ }
+ } else if !self.config.delete_after_read.unwrap_or(false)
+ && self.config.processed_column.is_none()
+ {
+ return Err(Error::InitError(format!(
+ "custom_query must filter with $offset (`WHERE {tracking_column} > $offset`), or \
+ set delete_after_read or processed_column"
+ )));
+ }
+
+ if query.contains("$table") && self.config.tables.is_empty() {
+ return Err(Error::InitError(
+ "custom_query uses $table but no tables are configured".to_string(),
+ ));
+ }
+ // Without $table the same SQL is executed once per table, so identical rows
+ // would be emitted repeatedly, each carrying a different table's metadata,
+ // message ID, and tracking offset. Table identity is not recoverable from a
+ // static query, so reject the combination instead of guessing one.
+ if !query.contains("$table") && self.config.tables.len() > 1 {
+ return Err(Error::InitError(format!(
+ "custom_query has no $table placeholder but {} tables are configured ({}), use \
+ $table in the query or configure a single table",
+ self.config.tables.len(),
+ self.config.tables.join(", ")
+ )));
+ }
+ Ok(())
+ }
+
+ fn build_polling_query(
+ &self,
+ table: &str,
+ tracking_column: &str,
+ last_offset: &Option<String>,
+ batch_size: u32,
+ kind: OffsetKind,
+ ) -> Result<String, Error> {
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_tracking = quote_identifier(tracking_column)?;
+
+ let base_query = format!("SELECT * FROM {quoted_table}");
+
+ let mut conditions = Vec::new();
+
+ if let Some(offset) = self.effective_offset(last_offset) {
+ conditions.push(format!(
+ "{quoted_tracking} > {}",
+ format_offset_value(&offset, kind)
+ ));
+ }
+
+ if let Some(processed_col) = &self.config.processed_column {
+ let quoted_processed = quote_identifier(processed_col)?;
+ conditions.push(format!("{quoted_processed} = FALSE"));
+ }
+
+ let where_clause = if conditions.is_empty() {
+ String::new()
+ } else {
+ format!(" WHERE {}", conditions.join(" AND "))
+ };
+
+ let order_clause = format!(" ORDER BY {quoted_tracking} ASC");
+ let limit_clause = format!(" LIMIT {batch_size}");
+
+ Ok(format!(
+ "{base_query}{where_clause}{order_clause}{limit_clause}"
+ ))
+ }
+
+ fn get_max_retries(&self) -> u32 {
+ self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES)
+ }
+
+ async fn apply_pending_operations(
+ &self,
+ operations: Vec<PendingOperation>,
+ deadline: tokio::time::Instant,
+ ) -> Result<Vec<PendingOperation>, Error> {
+ let mut failed_operations = Vec::new();
+ let mut operations = operations.into_iter();
+ while let Some(operation) = operations.next() {
+ if tokio::time::Instant::now() >= deadline {
+ failed_operations.push(operation);
+ failed_operations.extend(operations);
+ break;
+ }
+ let result = tokio::time::timeout_at(deadline, async {
+ let pool = self.get_pool()?;
+ self.mark_or_delete_processed_rows(pool, &operation).await
+ })
+ .await;
+ match result {
+ Ok(Ok(())) => {
+ self.consecutive_cleanup_failures
+ .store(0, Ordering::Relaxed);
+ }
+ Ok(Err(error)) => {
+ self.record_cleanup_failure(&operation, &error)?;
+ failed_operations.push(operation);
+ }
+ Err(_) => {
+ self.record_cleanup_failure(
+ &operation,
+ "cleanup exceeded the 10s budget for one Ack",
+ )?;
+ failed_operations.push(operation);
+ }
+ }
+ }
+ Ok(failed_operations)
+ }
+
+ fn record_cleanup_failure(
+ &self,
+ operation: &PendingOperation,
+ error: impl std::fmt::Display,
+ ) -> Result<(), Error> {
+ let consecutive_failures = self
+ .consecutive_cleanup_failures
+ .fetch_add(1, Ordering::Relaxed)
+ .saturating_add(1);
+ error!(
+ "Failed to mark or delete {} delivered row(s) in table '{}' for {CONNECTOR_NAME} \
+ connector ID: {}, retrying on the next poll ({consecutive_failures} consecutive \
+ failures): {error}",
+ operation.ids.len(),
+ operation.table,
+ self.id
+ );
+ if consecutive_failures >= MAX_CONSECUTIVE_CLEANUP_FAILURES {
+ return Err(Error::Connection(format!(
+ "stopping {CONNECTOR_NAME} connector ID {} after {consecutive_failures} \
+ consecutive mark/delete failures; the cleanup stays in the saved state and \
+ runs again on restart",
+ self.id
+ )));
+ }
+ Ok(())
+ }
+
+ async fn mark_or_delete_processed_rows(
+ &self,
+ pool: &Pool<MySql>,
+ operation: &PendingOperation,
+ ) -> Result<(), Error> {
+ let PendingOperation { table, ids, target } = operation;
+ if ids.is_empty() {
+ return Ok(());
+ }
+
+ let quoted_table = quote_qualified_identifier(table)?;
+ let quoted_pk = quote_identifier(&target.key_column)?;
+ let placeholders = vec!["?"; ids.len()].join(", ");
+
+ let query = match &target.action {
+ CleanupAction::Delete => {
+ if self.verbose {
+ info!("Deleting {} processed rows from '{table}'", ids.len());
+ } else {
+ debug!("Deleting {} processed rows from '{table}'", ids.len());
+ }
+ format!("DELETE FROM {quoted_table} WHERE {quoted_pk} IN ({placeholders})")
+ }
+ CleanupAction::MarkProcessed { column } => {
+ let quoted_processed = quote_identifier(column)?;
+ if self.verbose {
+ info!("Marking {} rows as processed in '{table}'", ids.len());
+ } else {
+ debug!("Marking {} rows as processed in '{table}'", ids.len());
+ }
+ format!(
+ "UPDATE {quoted_table} SET {quoted_processed} = TRUE WHERE {quoted_pk} IN ({placeholders})"
+ )
+ }
+ };
+
+ // The query and its binds are rebuilt on every attempt because with_retry
+ // calls the closure once per try and each `execute` consumes the query.
+ with_retry(
+ || {
+ let mut prepared = sqlx::query(sqlx::AssertSqlSafe(query.as_str()));
+ for id in ids {
+ prepared = match id {
+ PkValue::Text(text) => prepared.bind(text),
+ PkValue::Bytes(bytes) => prepared.bind(bytes),
+ };
+ }
+ prepared.execute(pool)
+ },
+ self.get_max_retries(),
+ self.retry_delay.as_millis() as u64,
+ )
+ .await
+ .map(|_| ())
+ }
+
+ async fn poll_tables(&self) -> Result<PolledBatch, Error> {
+ let pool = self.get_pool()?;
+
+ let batch_size = self.config.batch_size.unwrap_or(1000);
+ let tracking_column = self.tracking_column();
+ let pk_column = self.primary_key_column();
+
+ let row_config = RowProcessingConfig {
+ table: "",
+ tracking_column,
+ pk_column,
+ payload_format: self.payload_format(),
+ payload_col: self.config.payload_column.as_deref().unwrap_or(""),
+ snake_case_columns: self.config.snake_case_columns.unwrap_or(false),
+ include_metadata: self.config.include_metadata.unwrap_or(true),
+ };
+
+ // Phase 1: fetch and process every table into its own buffer without any
+ // side effect. A `process_row` failure is deterministic (a row that cannot
+ // be decoded fails identically every cycle), so a failing table is logged
+ // with its resume offset and skipped rather than aborting the whole poll:
+ // nothing has been marked yet, its rows stay in MySQL, and the remaining
+ // tables still make progress. The operator sees exactly which table and
+ // offset to fix.
+ //
+ // An ordering violation is handled one step further. Retrying it would
+ // re-run the same query, re-decode the same rows, and hit the same failure
+ // forever, so the table is latched off instead of being retried each cycle.
+ let mut batches: Vec<TableBatch> = Vec::with_capacity(self.config.tables.len());
+ let mut any_batch_full = false;
+
+ let poll_number = self.polls.fetch_add(1, Ordering::Relaxed) + 1;
+ let poisoned = { self.poisoned_tables.lock().await.clone() };
+ let mut newly_poisoned: Vec<String> = Vec::new();
+
+ for table in &self.config.tables {
+ if poisoned.contains(table) {
+ continue;
+ }
+
+ // Get last offset with minimal lock time
+ let last_offset = {
+ let state = self.state.lock().await;
+ state.tracking_offsets.get(table).cloned()
+ };
+
+ match self
+ .fetch_table_batch(
+ table,
+ &row_config,
+ tracking_column,
+ batch_size,
+ &last_offset,
+ )
+ .await
+ {
+ Ok(batch) => {
+ if self.verbose {
+ info!("Fetched {} rows from table '{table}'", batch.messages.len());
+ } else {
+ debug!("Fetched {} rows from table '{table}'", batch.messages.len());
+ }
+ if !batch.messages.is_empty() {
+ self.cursor_regressions.lock().await.remove(table);
+ }
+ // A full batch means the LIMIT was hit, so more rows are waiting.
+ any_batch_full |= batch.messages.len() as u32 >= batch_size;
+ batches.push(batch);
+ }
+ Err(FetchError::Ordering(reason) | FetchError::MissingTrackingColumn(reason)) => {
+ error!(
+ "Table '{table}' at offset {}: {reason}. Table disabled until restart.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ newly_poisoned.push(table.clone());
+ }
+ Err(FetchError::CursorRegression(reason)) => {
+ // Cleared only by a batch that carried rows, so the run counts polls
+ // since the table last published anything. An idle poll produces as
+ // little as a rejected one does and must not read as recovery.
+ let consecutive = {
+ let mut regressions = self.cursor_regressions.lock().await;
+ let consecutive = regressions.entry(table.clone()).or_insert(0);
+ *consecutive += 1;
+ *consecutive
+ };
+ let offset = last_offset.as_deref().unwrap_or("<start>");
+ if consecutive >= CURSOR_REGRESSION_ERROR_THRESHOLD {
+ error!(
+ "Table '{table}' at offset {offset}: {reason}. Skipping this poll \
+ ({consecutive} consecutive polls)."
+ );
+ } else {
+ warn!("Table '{table}' at offset {offset}: {reason}. Skipping this poll.");
+ }
+ }
+ Err(FetchError::StaleOffsetKind(reason)) => {
+ warn!(
+ "Table '{table}' at offset {}: {reason}. Retrying on the next poll.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::UnusableTrackingValue(reason)) => {
+ error!(
+ "Table '{table}' at offset {}: {reason}. Retrying each poll until the row \
+ is fixed.",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ Err(FetchError::Other(error)) => {
+ error!(
+ "Failed to process table '{table}' at offset {}, skipping this cycle: {error}",
+ last_offset.as_deref().unwrap_or("<start>")
+ );
+ }
+ }
+ }
+
+ if !newly_poisoned.is_empty() {
+ let mut poisoned_tables = self.poisoned_tables.lock().await;
+ poisoned_tables.extend(newly_poisoned);
+ if self
+ .config
+ .tables
+ .iter()
+ .all(|table| poisoned_tables.contains(table))
+ {
+ error!(
+ "All configured tables are disabled because their batches cannot yield a \
+ cursor; {CONNECTOR_NAME} connector with ID: {} is idle until restarted",
+ self.id
+ );
+ }
+ }
+
+ // From the next poll on a disabled table is skipped before anything is logged,
+ // so the one line written when it was disabled can scroll out of the operator's
+ // log window and leave a dark connector looking healthy.
+ if poll_number.is_multiple_of(DISABLED_TABLES_LOG_INTERVAL) {
+ let poisoned_tables = self.poisoned_tables.lock().await;
+ if !poisoned_tables.is_empty() {
+ let mut disabled: Vec<&str> = poisoned_tables.iter().map(String::as_str).collect();
+ disabled.sort_unstable();
+ error!(
+ "{} of {} configured table(s) are still disabled and are not being polled by \
+ {CONNECTOR_NAME} connector with ID: {}, and will stay that way until it is \
+ restarted: {}",
+ disabled.len(),
+ self.config.tables.len(),
+ self.id,
+ disabled.join(", ")
+ );
+ }
+ }
+
+ // Phase 2: stage each table's batch. Nothing touches MySQL or the live state
+ // here: the offsets and the mark/delete are applied in `on_batch_result` once
+ // the runtime has sent the batch and saved this candidate state, and a Nack
+ // throws the candidate away so the same rows are read again.
+ let cleanup_target = self.cleanup_target();
+ let mut candidate_state = self.state.lock().await.clone();
+ let mut operations = Vec::new();
+ let mut messages = Vec::new();
+ let mut total_processed: u64 = 0;
+
+ for mut batch in batches {
+ if let Some(target) = &cleanup_target
+ && !batch.messages.is_empty()
+ {
+ if batch.processed_ids.is_empty() {
+ error!(
+ "Table '{}': mark/delete is configured but no primary keys were extracted \
+ from {} row(s), skipping this cycle so rows are not published without \
+ being marked or deleted (check that '{pk_column}' is projected and scalar)",
+ batch.table,
+ batch.messages.len()
+ );
+ continue;
+ }
+
+ match self
+ .cleanup_key_is_unique(pool, &batch.table, &target.key_column)
+ .await
+ {
+ Ok(true) => {}
+ Ok(false) => {
+ error!(
+ "{}. Table disabled until restart.",
+ not_unique_key_message(&batch.table, &target.key_column)
+ );
+ self.poisoned_tables.lock().await.insert(batch.table);
+ continue;
+ }
+ Err(error) => {
+ error!("Table '{}' skipped this cycle: {error}", batch.table);
+ continue;
+ }
+ }
+
+ operations.push(PendingOperation {
+ table: batch.table.clone(),
+ ids: std::mem::take(&mut batch.processed_ids),
+ target: target.clone(),
+ });
+ }
+
+ total_processed += batch.messages.len() as u64;
+ messages.append(&mut batch.messages);
+ if let Some(offset) = batch.max_offset {
+ candidate_state.tracking_offsets.insert(batch.table, offset);
+ }
+ }
+
+ let pending = (total_processed > 0).then(|| {
+ candidate_state.processed_rows += total_processed;
+ candidate_state.pending_operations = operations;
+ PendingBatch {
+ state: candidate_state,
+ acknowledged: false,
+ retiring: false,
+ }
+ });
+
+ self.last_batch_full
+ .store(any_batch_full, Ordering::Relaxed);
+ Ok(PolledBatch { messages, pending })
+ }
+
+ async fn fetch_table_batch(
+ &self,
+ table: &str,
+ row_config: &RowProcessingConfig<'_>,
+ tracking_column: &str,
+ batch_size: u32,
+ last_offset: &Option<String>,
+ ) -> Result<TableBatch, FetchError> {
+ let pool = self.get_pool()?;
+ let table_config = RowProcessingConfig {
+ table,
+ ..*row_config
+ };
+
+ let query_kind = self.tracking_kind(table).await;
+ let query = if let Some(custom_query) = &self.config.custom_query {
+ self.substitute_query_params(custom_query, table, last_offset, batch_size, query_kind)?
+ } else {
+ self.build_polling_query(table, tracking_column, last_offset, batch_size, query_kind)?
+ };
+
+ // Database I/O without holding the lock
+ let rows = with_retry(
+ || sqlx::query(sqlx::AssertSqlSafe(query.as_str())).fetch_all(pool),
Review Comment:
fixed. Added `.persistent(false)` so there is no cached statement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org