You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "jaroslawroszyk (via GitHub)" <gi...@apache.org> on 2025/09/25 16:23:38 UTC
[PR] feat(connectors/postgres-source): add CDC backend selector, keep buil… [iggy]
jaroslawroszyk opened a new pull request, #2197:
URL: https://github.com/apache/iggy/pull/2197
…tin CDC, and add tests
- Add `cdc_backend` config ("builtin" | "pg_replicate") and route in open/poll
- Keep builtin CDC via pg_logical_slot_get_changes
- Introduce `cdc_pg_replicate` feature flag (placeholder); selecting pg_replicate returns clear not-implemented error
- Revert/remove incomplete pg_replicate wiring and URL/ETL deps to restore clean builds
- Add unit tests for CDC message parsing (INSERT/UPDATE/DELETE) and polling query builder
- Ensure crate compiles and tests pass
Refs #1849
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors/postgres-source): add CDC backend selector, keep buil… [iggy]
Posted by "jaroslawroszyk (via GitHub)" <gi...@apache.org>.
jaroslawroszyk commented on code in PR #2197:
URL: https://github.com/apache/iggy/pull/2197#discussion_r2399951027
##########
core/connectors/sources/postgres_source/src/lib.rs:
##########
@@ -685,3 +720,160 @@ impl Source for PostgresSource {
Ok(())
}
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn test_build_polling_query_with_last_offset() {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors/postgres-source): add CDC backend selector, keep buil… [iggy]
Posted by "spetz (via GitHub)" <gi...@apache.org>.
spetz commented on code in PR #2197:
URL: https://github.com/apache/iggy/pull/2197#discussion_r2384247724
##########
core/connectors/sources/postgres_source/src/lib.rs:
##########
@@ -685,3 +720,160 @@ impl Source for PostgresSource {
Ok(())
}
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn test_build_polling_query_with_last_offset() {
+ let src = PostgresSource::new(
+ 1,
+ PostgresSourceConfig {
+ connection_string: "postgres://localhost/db".to_string(),
+ mode: "polling".to_string(),
+ tables: vec!["users".to_string()],
+ poll_interval: Some("5s".to_string()),
+ batch_size: Some(500),
+ tracking_column: Some("updated_at".to_string()),
+ initial_offset: None,
+ max_connections: None,
+ enable_wal_cdc: None,
+ custom_query: None,
+ snake_case_columns: None,
+ include_metadata: None,
+ replication_slot: None,
+ publication_name: None,
+ capture_operations: None,
+ cdc_backend: None,
+ },
+ None,
+ );
+ let q = src.build_polling_query("users", "updated_at", &Some("2024-01-01".to_string()), 500);
Review Comment:
Let's use more meaningful names e.g. `query`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors/postgres-source): add CDC backend selector, keep buil… [iggy]
Posted by "spetz (via GitHub)" <gi...@apache.org>.
spetz merged PR #2197:
URL: https://github.com/apache/iggy/pull/2197
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors/postgres-source): add CDC backend selector, keep buil… [iggy]
Posted by "jaroslawroszyk (via GitHub)" <gi...@apache.org>.
jaroslawroszyk commented on code in PR #2197:
URL: https://github.com/apache/iggy/pull/2197#discussion_r2399951948
##########
core/connectors/sources/postgres_source/src/lib.rs:
##########
@@ -685,3 +720,160 @@ impl Source for PostgresSource {
Ok(())
}
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn test_build_polling_query_with_last_offset() {
+ let src = PostgresSource::new(
+ 1,
+ PostgresSourceConfig {
+ connection_string: "postgres://localhost/db".to_string(),
+ mode: "polling".to_string(),
+ tables: vec!["users".to_string()],
+ poll_interval: Some("5s".to_string()),
+ batch_size: Some(500),
+ tracking_column: Some("updated_at".to_string()),
+ initial_offset: None,
+ max_connections: None,
+ enable_wal_cdc: None,
+ custom_query: None,
+ snake_case_columns: None,
+ include_metadata: None,
+ replication_slot: None,
+ publication_name: None,
+ capture_operations: None,
+ cdc_backend: None,
+ },
+ None,
+ );
+ let q = src.build_polling_query("users", "updated_at", &Some("2024-01-01".to_string()), 500);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors/postgres-source): add CDC backend selector, keep buil… [iggy]
Posted by "spetz (via GitHub)" <gi...@apache.org>.
spetz commented on code in PR #2197:
URL: https://github.com/apache/iggy/pull/2197#discussion_r2384247574
##########
core/connectors/sources/postgres_source/src/lib.rs:
##########
@@ -685,3 +720,160 @@ impl Source for PostgresSource {
Ok(())
}
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn test_build_polling_query_with_last_offset() {
Review Comment:
Let's stick to more BDD like names e.g. `given_last_offset_polling_query_should_be_built`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org