You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "ryerraguntla (via GitHub)" <gi...@apache.org> on 2026/06/20 01:36:50 UTC
[PR] Feat(connectors)/opensearch source connector [iggy]
ryerraguntla opened a new pull request, #3515:
URL: https://github.com/apache/iggy/pull/3515
**Which issue does this PR close?**
Closes #3503
**Rationale**
Iggy had no native way to ingest documents from OpenSearch clusters into Iggy streams. This connector fills that gap for observability pipelines, audit log forwarding, and data lake ingestion scenarios where OpenSearch is the source of truth.
**What changed?**
**Before**: no OpenSearch source connector existed. Users had to write custom consumers using the OpenSearch client to bridge documents into Iggy.
**After**: a new iggy_connector_opensearch_source cdylib plugin implements the Source trait and is loaded by the connectors runtime via FFI. Each poll() issues a search_after-paginated query on (timestamp_field, _id), publishes each hit's _source as a JSON ProducedMessage, and persists the cursor in ConnectorState.
**Module layout**
```
core/connectors/sources/opensearch_source/
├── Cargo.toml
├── README.md
├── dependencies.md
└── src/
├── lib.rs # Source trait, poll loop, search_after cursor, config, 16 unit tests
├── state_manager.rs # File-backed SourceState mirror, atomic rename protocol, 4 unit tests
└── http_tests.rs # Axum mock server tests (20 tests, no Docker required)
core/connectors/runtime/example_config/connectors/
└── opensearch_source.toml
core/integration/tests/connectors/opensearch/
├── mod.rs
├── opensearch_source.rs # 7 Docker-backed integration tests
├── opensearch_source_types.rs
└── fixtures/opensearch/ # Container start/stop, document indexing helpers
```
**Key implementation decisionsCursor mechanism** —
**search_after on (timestamp_field, _id) sorted ascending.**
Two sort keys give stable pagination when timestamps collide; _id is the tiebreaker. No Scroll API, PIT, or offset paging.
**State authority** — runtime **ConnectorState** (MessagePack) is authoritative. An optional file-backed JSON mirror (SourceState) is written atomically on close() using write-tmp → fdatasync → rename → dir-fsync. On restart, if a valid runtime state exists it takes priority and file state is not loaded. If state deserialization fails, open() returns InitError to prevent a silent cursor reset.
**Validation** at open() — timestamp_field is required and non-empty (validated before any network call). batch_size = 0 is rejected. File state config is validated if state.enabled = true.
**Error handling** — transient OpenSearch errors fail the poll with the error counter incremented. No circuit breaker or HTTP retry in this version (documented limitation). poll() still sleeps polling_interval on error so the runtime does not spin.
**At-least-once delivery** — cursor advances in memory before ConnectorState is persisted by the runtime. A crash after poll() returns but before the runtime writes state can re-emit the last batch. Documented in README and Limitations.
**Validation**
```
cargo fmt --all
cargo sort --no-format --workspace
cargo clippy -p iggy_connector_opensearch_source --all-targets -- -D warnings
cargo test -p iggy_connector_opensearch_source
./scripts/ci/taplo.sh
./scripts/ci/license-headers.sh
# Docker-backed integration (requires running iggy-server and iggy-connectors binaries):
env CARGO_BIN_EXE_iggy-server=... CARGO_BIN_EXE_iggy-connectors=... \
cargo test -p integration --test mod -- connectors::opensearch::opensearch_source
```
All 40 tests pass (16 unit + 4 state_manager unit + 20 axum HTTP mock + 7 Docker integration).
**Local Execution**
Passed
Pre-commit hooks ran
**AI Usage**
1. Tools used: Claude
2. Scope: scaffolding initial file structure and boilerplate; all logic reviewed and iterated manually
3. Verification: full local compilation, all tests executed, end-to-end tested with Docker OpenSearch + Iggy server + connector runtime
4. Can explain every line: yes
**Configuration reference**
```
type = "source"
key = "opensearch"
enabled = true
version = 0
name = "OpenSearch source"
path = "<BASE_DIR>/target/release/libiggy_connector_opensearch_source"
plugin_config_format = "json"
[[streams]]
stream = "opensearch_stream"
topic = "documents"
schema = "json"
batch_length = 100
linger_time = "5ms"
[plugin_config]
url = "http://localhost:9200"
index = "logs-*"
timestamp_field = "@timestamp" # required; must exist on every document
polling_interval = "10s" # default
batch_size = 100 # default
# username = "admin"
# password = "replace_with_secret"
# verbose_logging = false
# Optional: custom query DSL (default: match_all)
# [plugin_config.query]
# match = { "log.level" = "error" }
# Optional: file-backed state mirror (runtime msgpack is authoritative)
# [plugin_config.state]
# enabled = true
# storage_type = "file"
# state_id = "opensearch_logs_connector"
# [plugin_config.state.storage_config]
# base_path = "./connector_states"
```
<h2>Required Fields</h2>
Field | Type | Description
-- | -- | --
url | String | OpenSearch HTTP base URL
index | String | Index name or pattern (logs-* supported)
timestamp_field | String | Document field used as sort key and cursor; must exist on every document
</body></html>
<h2 class="text-text-100 mt-3 -mb-1 text-[1.125rem] font-bold" data-sourcepos="11:1-11:19;335-353">Optional Fields</h2>
<div class="overflow-x-auto w-full px-2 mb-6" data-sourcepos="13:1-19:92;355-829">
Field | Type | Default | Description
-- | -- | -- | --
polling_interval | humantime | "10s" | Sleep after each completed poll cycle
batch_size | usize | 100 | Documents per search request (min 1)
query | JSON | {"match_all":{}} | OpenSearch query DSL applied on every poll
username / password | String | — | HTTP Basic auth
verbose_logging | bool | false | Log per-poll counts at info! instead of debug!
</div>
**Poll cycle**
Each poll() call:Issues POST /{index}/_search sorted by (timestamp_field asc, _id asc) with optional search_after.Maps each hit's _source to a JSON ProducedMessage.Advances the search_after cursor to the sort tuple of the last successfully published hit.Returns ProducedMessages with serialized ConnectorState.Sleeps polling_interval.Hits missing a sort tuple or _source are skipped with warn!; they do not block the cursor.
**Known limitations**
1. At-least-once delivery — cursor advances before the runtime persists ConnectorState; last batch may be re-emitted on crash.
2. Single sequential reader — one search_after cursor, one batch per poll; no parallel shard workers.
3. search_after only — no Scroll API, PIT, or sliced export.
4. Backfill gap — documents with timestamp_field older than the current cursor are not read without a state reset.
5. _source-disabled documents skipped permanently.Single-node transport — no cluster sniffing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on PR #3515:
URL: https://github.com/apache/iggy/pull/3515#issuecomment-4895775797
@ryerraguntla sure - please fix these togglers if you can. we're trying to push the clustering forward, thus less attention on currently open PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #3515:
URL: https://github.com/apache/iggy/pull/3515#discussion_r3522546284
##########
core/connectors/sources/opensearch_source/src/lib.rs:
##########
@@ -0,0 +1,1135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::retry::{CircuitBreaker, parse_duration};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use opensearch::{
+ OpenSearch, SearchParts,
+ auth::Credentials,
+ http::{Url, transport::TransportBuilder},
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use serde_json::{Value, json};
+use std::sync::Arc;
+use std::time::Duration;
+use tokio::{sync::Mutex, time::sleep};
+use tracing::{debug, error, info, warn};
+
+mod retry;
+mod state_manager;
+use crate::retry::{
+ DEFAULT_CB_COOL_DOWN, DEFAULT_CB_THRESHOLD, DEFAULT_MAX_OPEN_RETRIES, DEFAULT_MAX_RETRIES,
+ DEFAULT_OPEN_RETRY_MAX_DELAY, DEFAULT_RETRY_DELAY, DEFAULT_RETRY_MAX_DELAY, RetryBackoff,
+ is_transient_status, normalized_max_attempts, sleep_before_retry,
+};
+use crate::state_manager::{SOURCE_STATE_VERSION, SourceState, validate_state_storage_config};
+
+source_connector!(OpenSearchSource);
+
+const CONNECTOR_NAME: &str = "OpenSearch source";
+const DEFAULT_POLLING_INTERVAL: &str = "10s";
+const DEFAULT_BATCH_SIZE: usize = 100;
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct State {
+ #[serde(default)]
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ #[serde(default, alias = "total_documents_fetched")]
+ total_documents_published: usize,
+ #[serde(default)]
+ poll_count: usize,
+ /// OpenSearch `search_after` tuple from the last hit in the previous batch.
+ #[serde(default)]
+ search_after: Option<Vec<Value>>,
+ #[serde(default)]
+ error_count: usize,
+ #[serde(default)]
+ last_error: Option<String>,
+ #[serde(default)]
+ processing_stats: ProcessingStats,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct ProcessingStats {
+ #[serde(default)]
+ total_bytes_processed: u64,
+ /// Running cumulative average over the connector's lifetime, persisted and accumulated
+ /// across restarts. Reflects long-term throughput baseline, not session-only average.
+ #[serde(default)]
+ avg_batch_processing_time_ms: f64,
+ #[serde(default)]
+ last_successful_poll: Option<DateTime<Utc>>,
+ #[serde(default)]
+ empty_polls_count: usize,
+ #[serde(default)]
+ successful_polls_count: usize,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct StateConfig {
+ #[serde(default)]
+ pub enabled: bool,
+ pub storage_type: Option<String>,
+ pub storage_config: Option<Value>,
+ pub state_id: Option<String>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct OpenSearchSourceConfig {
+ pub url: String,
+ pub index: String,
+ pub username: Option<String>,
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_optional_secret")]
+ pub password: Option<SecretString>,
+ pub query: Option<Value>,
+ pub polling_interval: Option<String>,
+ pub batch_size: Option<usize>,
+ pub timestamp_field: Option<String>,
+ #[serde(default)]
+ pub verbose_logging: bool,
+ pub state: Option<StateConfig>,
+ /// Total poll-phase attempt count (not retry count); 1 = no retries, 3 = 2 retries.
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+ pub retry_max_delay: Option<String>,
+ /// Total open-phase attempt count (not retry count); 1 = no retries, 5 = 4 retries.
+ pub max_open_retries: Option<u32>,
+ pub open_retry_max_delay: Option<String>,
+ pub circuit_breaker_threshold: Option<u32>,
+ pub circuit_breaker_cool_down: Option<String>,
+}
+
+#[derive(Debug)]
+pub struct OpenSearchSource {
+ id: u32,
+ config: OpenSearchSourceConfig,
+ client: Option<OpenSearch>,
+ polling_interval: Duration,
+ /// Pre-built query body (query + size + sort). Set once in `open()` after config validation.
+ /// `None` before `open()` and after a failed `open()`.
+ search_body_base: Option<Value>,
+ verbose: bool,
+ max_retries: u32,
+ retry_delay: Duration,
+ retry_max_delay: Duration,
+ max_open_retries: u32,
+ open_retry_max_delay: Duration,
+ circuit_breaker: Arc<CircuitBreaker>,
+ state: Mutex<State>,
+ /// `Some(cause)` when runtime state restore was rejected; `None` means restore succeeded.
+ state_restore_error: Option<String>,
+ /// True when `new()` restored a valid runtime `ConnectorState`. File mirror must not override it.
+ runtime_state_restored: bool,
+}
+
+struct SearchOutcome {
+ messages: Vec<ProducedMessage>,
+ search_after: Option<Vec<Value>>,
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ batch_bytes: u64,
+}
+
+impl OpenSearchSource {
+ pub fn new(id: u32, config: OpenSearchSourceConfig, state: Option<ConnectorState>) -> Self {
+ let polling_interval =
+ parse_duration(config.polling_interval.as_deref(), DEFAULT_POLLING_INTERVAL);
+ let verbose = config.verbose_logging;
+ let (restored_state, state_restore_error, runtime_state_restored) =
+ restore_state(id, state);
+
+ let cb_threshold = config
+ .circuit_breaker_threshold
+ .unwrap_or(DEFAULT_CB_THRESHOLD);
+ let cb_cool_down = parse_duration(
+ config.circuit_breaker_cool_down.as_deref(),
+ DEFAULT_CB_COOL_DOWN,
+ );
+ let circuit_breaker = Arc::new(CircuitBreaker::new(cb_threshold, cb_cool_down));
+ let max_retries =
+ normalized_max_attempts(config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES));
+ let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+ let retry_max_delay =
+ parse_duration(config.retry_max_delay.as_deref(), DEFAULT_RETRY_MAX_DELAY);
+ let max_open_retries =
+ normalized_max_attempts(config.max_open_retries.unwrap_or(DEFAULT_MAX_OPEN_RETRIES));
+ let open_retry_max_delay = parse_duration(
+ config.open_retry_max_delay.as_deref(),
+ DEFAULT_OPEN_RETRY_MAX_DELAY,
+ );
+
+ OpenSearchSource {
+ id,
+ config,
+ max_retries,
+ retry_delay,
+ retry_max_delay,
+ max_open_retries,
+ open_retry_max_delay,
+ circuit_breaker,
+ client: None,
+ polling_interval,
+ search_body_base: None,
+ verbose,
+ state: Mutex::new(restored_state),
+ state_restore_error,
+ runtime_state_restored,
+ }
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn batch_size(&self) -> usize {
+ self.config.batch_size.unwrap_or(DEFAULT_BATCH_SIZE)
+ }
+
+ fn timestamp_field(&self) -> Result<&str, Error> {
+ self.config
+ .timestamp_field
+ .as_deref()
+ .filter(|s| !s.is_empty())
+ .ok_or_else(|| Error::InvalidConfigValue("timestamp_field is required".to_string()))
+ }
+
+ fn get_state_id(&self) -> String {
+ self.config
+ .state
+ .as_ref()
+ .and_then(|s| s.state_id.clone())
+ .unwrap_or_else(|| format!("opensearch_source_{}", self.id))
+ }
+
+ pub(crate) async fn internal_state_to_source_state(&self) -> Result<SourceState, Error> {
+ let state = self.state.lock().await;
+ let data = serde_json::to_value(&*state).map_err(|error| {
+ Error::Serialization(format!("Failed to serialize connector state: {error}"))
+ })?;
+
+ Ok(SourceState {
+ id: self.get_state_id(),
+ last_updated: Utc::now(),
+ version: SOURCE_STATE_VERSION,
+ data,
+ metadata: Some(json!({
+ "connector_type": "opensearch_source",
+ "connector_id": self.id,
+ "index": self.config.index,
+ })),
+ })
+ }
+
+ pub(crate) async fn source_state_to_internal_state(
+ &mut self,
+ source_state: SourceState,
+ ) -> Result<(), Error> {
+ if source_state.version != SOURCE_STATE_VERSION {
+ return Err(Error::Serialization(format!(
+ "unsupported file state version {}, expected {SOURCE_STATE_VERSION}",
+ source_state.version
+ )));
+ }
+
+ let restored: State = serde_json::from_value(source_state.data).map_err(|error| {
+ Error::Serialization(format!("Failed to deserialize connector state: {error}"))
+ })?;
+
+ let mut state = self.state.lock().await;
+ *state = restored;
+ Ok(())
+ }
+
+ async fn create_client(&self) -> Result<OpenSearch, Error> {
+ let url = Url::parse(&self.config.url).map_err(|error| {
+ Error::InvalidConfigValue(format!("Invalid OpenSearch URL: {error}"))
+ })?;
+
+ let conn_pool = opensearch::http::transport::SingleNodeConnectionPool::new(url);
+ let mut transport_builder = TransportBuilder::new(conn_pool);
+
+ if let (Some(username), Some(password)) = (&self.config.username, &self.config.password) {
+ let credentials =
+ Credentials::Basic(username.clone(), password.expose_secret().to_string());
+ transport_builder = transport_builder.auth(credentials);
+ }
+
+ let transport = transport_builder
+ .build()
+ .map_err(|error| Error::InitError(format!("Failed to build transport: {error}")))?;
+
+ Ok(OpenSearch::new(transport))
+ }
+
+ async fn check_index_exists_with_retry(&self, client: &OpenSearch) -> Result<(), Error> {
+ let max_attempts = self.max_open_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let response = match client
+ .indices()
+ .exists(opensearch::indices::IndicesExistsParts::Index(&[&self
+ .config
+ .index]))
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ // retry_delay is shared with the poll phase (no separate open-phase
+ // base delay); open_retry_max_delay caps the growth independently.
+ // Same pattern as the InfluxDB source connector.
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::InitError(format!(
+ "Failed to check index existence: {error}"
+ )));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(());
+ }
+
+ let status = response.status_code().as_u16();
+ if status == 404 {
+ return Err(Error::InitError(format!(
+ "Index '{}' does not exist or is not accessible",
+ self.config.index
+ )));
+ }
+
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::InitError(format!(
+ "Index '{}' check failed: HTTP {status}: {error_text}",
+ self.config.index
+ )));
+ }
+ }
+
+ async fn send_search_with_retry(
+ &self,
+ client: &OpenSearch,
+ mut search_body: Value,
+ ) -> Result<opensearch::http::response::Response, Error> {
+ let max_attempts = self.max_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let body = if attempt < max_attempts {
+ search_body.clone()
+ } else {
+ std::mem::take(&mut search_body)
+ };
+ let response = match client
+ .search(SearchParts::Index(&[&self.config.index]))
+ .body(body)
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::Storage(format!("Failed to execute search: {error}")));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(response);
+ }
+
+ let status = response.status_code().as_u16();
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::Storage(format!(
+ "Search request failed: {error_text}"
+ )));
+ }
+ }
+
+ async fn search_documents(&self, client: &OpenSearch) -> Result<SearchOutcome, Error> {
+ let state = self.state.lock().await;
+ let search_after = state.search_after.clone();
+ drop(state);
+
+ let timestamp_field = self.timestamp_field()?;
+
+ let mut search_body = self
+ .search_body_base
+ .as_ref()
+ .ok_or_else(|| {
+ Error::Connection("connector not initialized; call open() first".to_string())
+ })?
+ .clone();
+
+ if let Some(cursor) = search_after {
+ search_body["search_after"] = json!(cursor);
+ }
+
+ let response = self.send_search_with_retry(client, search_body).await?;
+
+ let mut response_body: Value = response
+ .json()
+ .await
+ .map_err(|e| Error::Storage(format!("Failed to parse search response: {e}")))?;
+
+ let hits: Vec<Value> = response_body
+ .get_mut("hits")
+ .and_then(|h| h.get_mut("hits"))
+ .and_then(|arr| arr.as_array_mut())
+ .map(std::mem::take)
+ .unwrap_or_default();
+
+ let mut messages = Vec::with_capacity(hits.len());
+ let mut batch_bytes = 0u64;
+ let mut last_sort = None;
+ let mut last_poll_timestamp = None;
+
+ for hit in &hits {
+ let Some(sort) = hit
+ .get("sort")
+ .and_then(|s| s.as_array())
+ .filter(|a| !a.is_empty())
+ else {
+ warn!(
+ connector_id = self.id,
+ hit_id = hit.get("_id").and_then(|value| value.as_str()),
+ "Skipping OpenSearch hit without sort tuple; document will not be published"
+ );
+ continue;
+ };
+
+ last_sort = Some(sort);
Review Comment:
Another good one. A common issue across many reviewed connectors. Took care of it and adding test cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #3515:
URL: https://github.com/apache/iggy/pull/3515#discussion_r3530518819
##########
core/connectors/sources/opensearch_source/src/lib.rs:
##########
@@ -0,0 +1,1135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::retry::{CircuitBreaker, parse_duration};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use opensearch::{
+ OpenSearch, SearchParts,
+ auth::Credentials,
+ http::{Url, transport::TransportBuilder},
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use serde_json::{Value, json};
+use std::sync::Arc;
+use std::time::Duration;
+use tokio::{sync::Mutex, time::sleep};
+use tracing::{debug, error, info, warn};
+
+mod retry;
+mod state_manager;
+use crate::retry::{
+ DEFAULT_CB_COOL_DOWN, DEFAULT_CB_THRESHOLD, DEFAULT_MAX_OPEN_RETRIES, DEFAULT_MAX_RETRIES,
+ DEFAULT_OPEN_RETRY_MAX_DELAY, DEFAULT_RETRY_DELAY, DEFAULT_RETRY_MAX_DELAY, RetryBackoff,
+ is_transient_status, normalized_max_attempts, sleep_before_retry,
+};
+use crate::state_manager::{SOURCE_STATE_VERSION, SourceState, validate_state_storage_config};
+
+source_connector!(OpenSearchSource);
+
+const CONNECTOR_NAME: &str = "OpenSearch source";
+const DEFAULT_POLLING_INTERVAL: &str = "10s";
+const DEFAULT_BATCH_SIZE: usize = 100;
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct State {
+ #[serde(default)]
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ #[serde(default, alias = "total_documents_fetched")]
+ total_documents_published: usize,
+ #[serde(default)]
+ poll_count: usize,
+ /// OpenSearch `search_after` tuple from the last hit in the previous batch.
+ #[serde(default)]
+ search_after: Option<Vec<Value>>,
+ #[serde(default)]
+ error_count: usize,
+ #[serde(default)]
+ last_error: Option<String>,
+ #[serde(default)]
+ processing_stats: ProcessingStats,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct ProcessingStats {
+ #[serde(default)]
+ total_bytes_processed: u64,
+ /// Running cumulative average over the connector's lifetime, persisted and accumulated
+ /// across restarts. Reflects long-term throughput baseline, not session-only average.
+ #[serde(default)]
+ avg_batch_processing_time_ms: f64,
+ #[serde(default)]
+ last_successful_poll: Option<DateTime<Utc>>,
+ #[serde(default)]
+ empty_polls_count: usize,
+ #[serde(default)]
+ successful_polls_count: usize,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct StateConfig {
+ #[serde(default)]
+ pub enabled: bool,
+ pub storage_type: Option<String>,
+ pub storage_config: Option<Value>,
+ pub state_id: Option<String>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct OpenSearchSourceConfig {
+ pub url: String,
+ pub index: String,
+ pub username: Option<String>,
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_optional_secret")]
+ pub password: Option<SecretString>,
+ pub query: Option<Value>,
+ pub polling_interval: Option<String>,
+ pub batch_size: Option<usize>,
+ pub timestamp_field: Option<String>,
+ #[serde(default)]
+ pub verbose_logging: bool,
+ pub state: Option<StateConfig>,
+ /// Total poll-phase attempt count (not retry count); 1 = no retries, 3 = 2 retries.
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+ pub retry_max_delay: Option<String>,
+ /// Total open-phase attempt count (not retry count); 1 = no retries, 5 = 4 retries.
+ pub max_open_retries: Option<u32>,
+ pub open_retry_max_delay: Option<String>,
+ pub circuit_breaker_threshold: Option<u32>,
+ pub circuit_breaker_cool_down: Option<String>,
+}
+
+#[derive(Debug)]
+pub struct OpenSearchSource {
+ id: u32,
+ config: OpenSearchSourceConfig,
+ client: Option<OpenSearch>,
+ polling_interval: Duration,
+ /// Pre-built query body (query + size + sort). Set once in `open()` after config validation.
+ /// `None` before `open()` and after a failed `open()`.
+ search_body_base: Option<Value>,
+ verbose: bool,
+ max_retries: u32,
+ retry_delay: Duration,
+ retry_max_delay: Duration,
+ max_open_retries: u32,
+ open_retry_max_delay: Duration,
+ circuit_breaker: Arc<CircuitBreaker>,
+ state: Mutex<State>,
+ /// `Some(cause)` when runtime state restore was rejected; `None` means restore succeeded.
+ state_restore_error: Option<String>,
+ /// True when `new()` restored a valid runtime `ConnectorState`. File mirror must not override it.
+ runtime_state_restored: bool,
+}
+
+struct SearchOutcome {
+ messages: Vec<ProducedMessage>,
+ search_after: Option<Vec<Value>>,
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ batch_bytes: u64,
+}
+
+impl OpenSearchSource {
+ pub fn new(id: u32, config: OpenSearchSourceConfig, state: Option<ConnectorState>) -> Self {
+ let polling_interval =
+ parse_duration(config.polling_interval.as_deref(), DEFAULT_POLLING_INTERVAL);
+ let verbose = config.verbose_logging;
+ let (restored_state, state_restore_error, runtime_state_restored) =
+ restore_state(id, state);
+
+ let cb_threshold = config
+ .circuit_breaker_threshold
+ .unwrap_or(DEFAULT_CB_THRESHOLD);
+ let cb_cool_down = parse_duration(
+ config.circuit_breaker_cool_down.as_deref(),
+ DEFAULT_CB_COOL_DOWN,
+ );
+ let circuit_breaker = Arc::new(CircuitBreaker::new(cb_threshold, cb_cool_down));
+ let max_retries =
+ normalized_max_attempts(config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES));
+ let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+ let retry_max_delay =
+ parse_duration(config.retry_max_delay.as_deref(), DEFAULT_RETRY_MAX_DELAY);
+ let max_open_retries =
+ normalized_max_attempts(config.max_open_retries.unwrap_or(DEFAULT_MAX_OPEN_RETRIES));
+ let open_retry_max_delay = parse_duration(
+ config.open_retry_max_delay.as_deref(),
+ DEFAULT_OPEN_RETRY_MAX_DELAY,
+ );
+
+ OpenSearchSource {
+ id,
+ config,
+ max_retries,
+ retry_delay,
+ retry_max_delay,
+ max_open_retries,
+ open_retry_max_delay,
+ circuit_breaker,
+ client: None,
+ polling_interval,
+ search_body_base: None,
+ verbose,
+ state: Mutex::new(restored_state),
+ state_restore_error,
+ runtime_state_restored,
+ }
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn batch_size(&self) -> usize {
+ self.config.batch_size.unwrap_or(DEFAULT_BATCH_SIZE)
+ }
+
+ fn timestamp_field(&self) -> Result<&str, Error> {
+ self.config
+ .timestamp_field
+ .as_deref()
+ .filter(|s| !s.is_empty())
+ .ok_or_else(|| Error::InvalidConfigValue("timestamp_field is required".to_string()))
+ }
+
+ fn get_state_id(&self) -> String {
+ self.config
+ .state
+ .as_ref()
+ .and_then(|s| s.state_id.clone())
+ .unwrap_or_else(|| format!("opensearch_source_{}", self.id))
+ }
+
+ pub(crate) async fn internal_state_to_source_state(&self) -> Result<SourceState, Error> {
+ let state = self.state.lock().await;
+ let data = serde_json::to_value(&*state).map_err(|error| {
+ Error::Serialization(format!("Failed to serialize connector state: {error}"))
+ })?;
+
+ Ok(SourceState {
+ id: self.get_state_id(),
+ last_updated: Utc::now(),
+ version: SOURCE_STATE_VERSION,
+ data,
+ metadata: Some(json!({
+ "connector_type": "opensearch_source",
+ "connector_id": self.id,
+ "index": self.config.index,
+ })),
+ })
+ }
+
+ pub(crate) async fn source_state_to_internal_state(
+ &mut self,
+ source_state: SourceState,
+ ) -> Result<(), Error> {
+ if source_state.version != SOURCE_STATE_VERSION {
+ return Err(Error::Serialization(format!(
+ "unsupported file state version {}, expected {SOURCE_STATE_VERSION}",
+ source_state.version
+ )));
+ }
+
+ let restored: State = serde_json::from_value(source_state.data).map_err(|error| {
+ Error::Serialization(format!("Failed to deserialize connector state: {error}"))
+ })?;
+
+ let mut state = self.state.lock().await;
+ *state = restored;
+ Ok(())
+ }
+
+ async fn create_client(&self) -> Result<OpenSearch, Error> {
+ let url = Url::parse(&self.config.url).map_err(|error| {
+ Error::InvalidConfigValue(format!("Invalid OpenSearch URL: {error}"))
+ })?;
+
+ let conn_pool = opensearch::http::transport::SingleNodeConnectionPool::new(url);
+ let mut transport_builder = TransportBuilder::new(conn_pool);
+
+ if let (Some(username), Some(password)) = (&self.config.username, &self.config.password) {
+ let credentials =
+ Credentials::Basic(username.clone(), password.expose_secret().to_string());
+ transport_builder = transport_builder.auth(credentials);
+ }
+
+ let transport = transport_builder
+ .build()
+ .map_err(|error| Error::InitError(format!("Failed to build transport: {error}")))?;
+
+ Ok(OpenSearch::new(transport))
+ }
+
+ async fn check_index_exists_with_retry(&self, client: &OpenSearch) -> Result<(), Error> {
+ let max_attempts = self.max_open_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let response = match client
+ .indices()
+ .exists(opensearch::indices::IndicesExistsParts::Index(&[&self
+ .config
+ .index]))
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ // retry_delay is shared with the poll phase (no separate open-phase
+ // base delay); open_retry_max_delay caps the growth independently.
+ // Same pattern as the InfluxDB source connector.
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::InitError(format!(
+ "Failed to check index existence: {error}"
+ )));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(());
+ }
+
+ let status = response.status_code().as_u16();
+ if status == 404 {
+ return Err(Error::InitError(format!(
+ "Index '{}' does not exist or is not accessible",
+ self.config.index
+ )));
+ }
+
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::InitError(format!(
+ "Index '{}' check failed: HTTP {status}: {error_text}",
+ self.config.index
+ )));
+ }
+ }
+
+ async fn send_search_with_retry(
+ &self,
+ client: &OpenSearch,
+ mut search_body: Value,
+ ) -> Result<opensearch::http::response::Response, Error> {
+ let max_attempts = self.max_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let body = if attempt < max_attempts {
+ search_body.clone()
+ } else {
+ std::mem::take(&mut search_body)
+ };
+ let response = match client
+ .search(SearchParts::Index(&[&self.config.index]))
+ .body(body)
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::Storage(format!("Failed to execute search: {error}")));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(response);
+ }
+
+ let status = response.status_code().as_u16();
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::Storage(format!(
+ "Search request failed: {error_text}"
+ )));
+ }
+ }
+
+ async fn search_documents(&self, client: &OpenSearch) -> Result<SearchOutcome, Error> {
+ let state = self.state.lock().await;
+ let search_after = state.search_after.clone();
+ drop(state);
+
+ let timestamp_field = self.timestamp_field()?;
+
+ let mut search_body = self
+ .search_body_base
+ .as_ref()
+ .ok_or_else(|| {
+ Error::Connection("connector not initialized; call open() first".to_string())
+ })?
+ .clone();
+
+ if let Some(cursor) = search_after {
+ search_body["search_after"] = json!(cursor);
+ }
+
+ let response = self.send_search_with_retry(client, search_body).await?;
+
+ let mut response_body: Value = response
+ .json()
+ .await
+ .map_err(|e| Error::Storage(format!("Failed to parse search response: {e}")))?;
+
+ let hits: Vec<Value> = response_body
+ .get_mut("hits")
+ .and_then(|h| h.get_mut("hits"))
+ .and_then(|arr| arr.as_array_mut())
+ .map(std::mem::take)
+ .unwrap_or_default();
+
+ let mut messages = Vec::with_capacity(hits.len());
+ let mut batch_bytes = 0u64;
+ let mut last_sort = None;
+ let mut last_poll_timestamp = None;
+
+ for hit in &hits {
+ let Some(sort) = hit
+ .get("sort")
+ .and_then(|s| s.as_array())
+ .filter(|a| !a.is_empty())
+ else {
+ warn!(
+ connector_id = self.id,
+ hit_id = hit.get("_id").and_then(|value| value.as_str()),
+ "Skipping OpenSearch hit without sort tuple; document will not be published"
+ );
+ continue;
+ };
+
+ last_sort = Some(sort);
Review Comment:
how would the test case help - I am habituated with writing the use cases, test cases then start writing the code - TDD. Thinking of different data set scenarios and mix of data is one of the criteria in the Igyy connectors development. This is the reason, my source contributions have a higher percentage of code coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "pushp-ranja (via GitHub)" <gi...@apache.org>.
pushp-ranja commented on code in PR #3515:
URL: https://github.com/apache/iggy/pull/3515#discussion_r3524933218
##########
core/connectors/sources/opensearch_source/src/lib.rs:
##########
@@ -0,0 +1,1135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::retry::{CircuitBreaker, parse_duration};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use opensearch::{
+ OpenSearch, SearchParts,
+ auth::Credentials,
+ http::{Url, transport::TransportBuilder},
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use serde_json::{Value, json};
+use std::sync::Arc;
+use std::time::Duration;
+use tokio::{sync::Mutex, time::sleep};
+use tracing::{debug, error, info, warn};
+
+mod retry;
+mod state_manager;
+use crate::retry::{
+ DEFAULT_CB_COOL_DOWN, DEFAULT_CB_THRESHOLD, DEFAULT_MAX_OPEN_RETRIES, DEFAULT_MAX_RETRIES,
+ DEFAULT_OPEN_RETRY_MAX_DELAY, DEFAULT_RETRY_DELAY, DEFAULT_RETRY_MAX_DELAY, RetryBackoff,
+ is_transient_status, normalized_max_attempts, sleep_before_retry,
+};
+use crate::state_manager::{SOURCE_STATE_VERSION, SourceState, validate_state_storage_config};
+
+source_connector!(OpenSearchSource);
+
+const CONNECTOR_NAME: &str = "OpenSearch source";
+const DEFAULT_POLLING_INTERVAL: &str = "10s";
+const DEFAULT_BATCH_SIZE: usize = 100;
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct State {
+ #[serde(default)]
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ #[serde(default, alias = "total_documents_fetched")]
+ total_documents_published: usize,
+ #[serde(default)]
+ poll_count: usize,
+ /// OpenSearch `search_after` tuple from the last hit in the previous batch.
+ #[serde(default)]
+ search_after: Option<Vec<Value>>,
+ #[serde(default)]
+ error_count: usize,
+ #[serde(default)]
+ last_error: Option<String>,
+ #[serde(default)]
+ processing_stats: ProcessingStats,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct ProcessingStats {
+ #[serde(default)]
+ total_bytes_processed: u64,
+ /// Running cumulative average over the connector's lifetime, persisted and accumulated
+ /// across restarts. Reflects long-term throughput baseline, not session-only average.
+ #[serde(default)]
+ avg_batch_processing_time_ms: f64,
+ #[serde(default)]
+ last_successful_poll: Option<DateTime<Utc>>,
+ #[serde(default)]
+ empty_polls_count: usize,
+ #[serde(default)]
+ successful_polls_count: usize,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct StateConfig {
+ #[serde(default)]
+ pub enabled: bool,
+ pub storage_type: Option<String>,
+ pub storage_config: Option<Value>,
+ pub state_id: Option<String>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct OpenSearchSourceConfig {
+ pub url: String,
+ pub index: String,
+ pub username: Option<String>,
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_optional_secret")]
+ pub password: Option<SecretString>,
+ pub query: Option<Value>,
+ pub polling_interval: Option<String>,
+ pub batch_size: Option<usize>,
+ pub timestamp_field: Option<String>,
+ #[serde(default)]
+ pub verbose_logging: bool,
+ pub state: Option<StateConfig>,
+ /// Total poll-phase attempt count (not retry count); 1 = no retries, 3 = 2 retries.
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+ pub retry_max_delay: Option<String>,
+ /// Total open-phase attempt count (not retry count); 1 = no retries, 5 = 4 retries.
+ pub max_open_retries: Option<u32>,
+ pub open_retry_max_delay: Option<String>,
+ pub circuit_breaker_threshold: Option<u32>,
+ pub circuit_breaker_cool_down: Option<String>,
+}
+
+#[derive(Debug)]
+pub struct OpenSearchSource {
+ id: u32,
+ config: OpenSearchSourceConfig,
+ client: Option<OpenSearch>,
+ polling_interval: Duration,
+ /// Pre-built query body (query + size + sort). Set once in `open()` after config validation.
+ /// `None` before `open()` and after a failed `open()`.
+ search_body_base: Option<Value>,
+ verbose: bool,
+ max_retries: u32,
+ retry_delay: Duration,
+ retry_max_delay: Duration,
+ max_open_retries: u32,
+ open_retry_max_delay: Duration,
+ circuit_breaker: Arc<CircuitBreaker>,
+ state: Mutex<State>,
+ /// `Some(cause)` when runtime state restore was rejected; `None` means restore succeeded.
+ state_restore_error: Option<String>,
+ /// True when `new()` restored a valid runtime `ConnectorState`. File mirror must not override it.
+ runtime_state_restored: bool,
+}
+
+struct SearchOutcome {
+ messages: Vec<ProducedMessage>,
+ search_after: Option<Vec<Value>>,
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ batch_bytes: u64,
+}
+
+impl OpenSearchSource {
+ pub fn new(id: u32, config: OpenSearchSourceConfig, state: Option<ConnectorState>) -> Self {
+ let polling_interval =
+ parse_duration(config.polling_interval.as_deref(), DEFAULT_POLLING_INTERVAL);
+ let verbose = config.verbose_logging;
+ let (restored_state, state_restore_error, runtime_state_restored) =
+ restore_state(id, state);
+
+ let cb_threshold = config
+ .circuit_breaker_threshold
+ .unwrap_or(DEFAULT_CB_THRESHOLD);
+ let cb_cool_down = parse_duration(
+ config.circuit_breaker_cool_down.as_deref(),
+ DEFAULT_CB_COOL_DOWN,
+ );
+ let circuit_breaker = Arc::new(CircuitBreaker::new(cb_threshold, cb_cool_down));
+ let max_retries =
+ normalized_max_attempts(config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES));
+ let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+ let retry_max_delay =
+ parse_duration(config.retry_max_delay.as_deref(), DEFAULT_RETRY_MAX_DELAY);
+ let max_open_retries =
+ normalized_max_attempts(config.max_open_retries.unwrap_or(DEFAULT_MAX_OPEN_RETRIES));
+ let open_retry_max_delay = parse_duration(
+ config.open_retry_max_delay.as_deref(),
+ DEFAULT_OPEN_RETRY_MAX_DELAY,
+ );
+
+ OpenSearchSource {
+ id,
+ config,
+ max_retries,
+ retry_delay,
+ retry_max_delay,
+ max_open_retries,
+ open_retry_max_delay,
+ circuit_breaker,
+ client: None,
+ polling_interval,
+ search_body_base: None,
+ verbose,
+ state: Mutex::new(restored_state),
+ state_restore_error,
+ runtime_state_restored,
+ }
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn batch_size(&self) -> usize {
+ self.config.batch_size.unwrap_or(DEFAULT_BATCH_SIZE)
+ }
+
+ fn timestamp_field(&self) -> Result<&str, Error> {
+ self.config
+ .timestamp_field
+ .as_deref()
+ .filter(|s| !s.is_empty())
+ .ok_or_else(|| Error::InvalidConfigValue("timestamp_field is required".to_string()))
+ }
+
+ fn get_state_id(&self) -> String {
+ self.config
+ .state
+ .as_ref()
+ .and_then(|s| s.state_id.clone())
+ .unwrap_or_else(|| format!("opensearch_source_{}", self.id))
+ }
+
+ pub(crate) async fn internal_state_to_source_state(&self) -> Result<SourceState, Error> {
+ let state = self.state.lock().await;
+ let data = serde_json::to_value(&*state).map_err(|error| {
+ Error::Serialization(format!("Failed to serialize connector state: {error}"))
+ })?;
+
+ Ok(SourceState {
+ id: self.get_state_id(),
+ last_updated: Utc::now(),
+ version: SOURCE_STATE_VERSION,
+ data,
+ metadata: Some(json!({
+ "connector_type": "opensearch_source",
+ "connector_id": self.id,
+ "index": self.config.index,
+ })),
+ })
+ }
+
+ pub(crate) async fn source_state_to_internal_state(
+ &mut self,
+ source_state: SourceState,
+ ) -> Result<(), Error> {
+ if source_state.version != SOURCE_STATE_VERSION {
+ return Err(Error::Serialization(format!(
+ "unsupported file state version {}, expected {SOURCE_STATE_VERSION}",
+ source_state.version
+ )));
+ }
+
+ let restored: State = serde_json::from_value(source_state.data).map_err(|error| {
+ Error::Serialization(format!("Failed to deserialize connector state: {error}"))
+ })?;
+
+ let mut state = self.state.lock().await;
+ *state = restored;
+ Ok(())
+ }
+
+ async fn create_client(&self) -> Result<OpenSearch, Error> {
+ let url = Url::parse(&self.config.url).map_err(|error| {
+ Error::InvalidConfigValue(format!("Invalid OpenSearch URL: {error}"))
+ })?;
+
+ let conn_pool = opensearch::http::transport::SingleNodeConnectionPool::new(url);
+ let mut transport_builder = TransportBuilder::new(conn_pool);
+
+ if let (Some(username), Some(password)) = (&self.config.username, &self.config.password) {
+ let credentials =
+ Credentials::Basic(username.clone(), password.expose_secret().to_string());
+ transport_builder = transport_builder.auth(credentials);
+ }
+
+ let transport = transport_builder
+ .build()
+ .map_err(|error| Error::InitError(format!("Failed to build transport: {error}")))?;
+
+ Ok(OpenSearch::new(transport))
+ }
+
+ async fn check_index_exists_with_retry(&self, client: &OpenSearch) -> Result<(), Error> {
+ let max_attempts = self.max_open_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let response = match client
+ .indices()
+ .exists(opensearch::indices::IndicesExistsParts::Index(&[&self
+ .config
+ .index]))
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ // retry_delay is shared with the poll phase (no separate open-phase
+ // base delay); open_retry_max_delay caps the growth independently.
+ // Same pattern as the InfluxDB source connector.
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::InitError(format!(
+ "Failed to check index existence: {error}"
+ )));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(());
+ }
+
+ let status = response.status_code().as_u16();
+ if status == 404 {
+ return Err(Error::InitError(format!(
+ "Index '{}' does not exist or is not accessible",
+ self.config.index
+ )));
+ }
+
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::InitError(format!(
+ "Index '{}' check failed: HTTP {status}: {error_text}",
+ self.config.index
+ )));
+ }
+ }
+
+ async fn send_search_with_retry(
+ &self,
+ client: &OpenSearch,
+ mut search_body: Value,
+ ) -> Result<opensearch::http::response::Response, Error> {
+ let max_attempts = self.max_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let body = if attempt < max_attempts {
+ search_body.clone()
+ } else {
+ std::mem::take(&mut search_body)
+ };
+ let response = match client
+ .search(SearchParts::Index(&[&self.config.index]))
+ .body(body)
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::Storage(format!("Failed to execute search: {error}")));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(response);
+ }
+
+ let status = response.status_code().as_u16();
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::Storage(format!(
+ "Search request failed: {error_text}"
+ )));
+ }
+ }
+
+ async fn search_documents(&self, client: &OpenSearch) -> Result<SearchOutcome, Error> {
+ let state = self.state.lock().await;
+ let search_after = state.search_after.clone();
+ drop(state);
+
+ let timestamp_field = self.timestamp_field()?;
+
+ let mut search_body = self
+ .search_body_base
+ .as_ref()
+ .ok_or_else(|| {
+ Error::Connection("connector not initialized; call open() first".to_string())
+ })?
+ .clone();
+
+ if let Some(cursor) = search_after {
+ search_body["search_after"] = json!(cursor);
+ }
+
+ let response = self.send_search_with_retry(client, search_body).await?;
+
+ let mut response_body: Value = response
+ .json()
+ .await
+ .map_err(|e| Error::Storage(format!("Failed to parse search response: {e}")))?;
+
+ let hits: Vec<Value> = response_body
+ .get_mut("hits")
+ .and_then(|h| h.get_mut("hits"))
+ .and_then(|arr| arr.as_array_mut())
+ .map(std::mem::take)
+ .unwrap_or_default();
+
+ let mut messages = Vec::with_capacity(hits.len());
+ let mut batch_bytes = 0u64;
+ let mut last_sort = None;
+ let mut last_poll_timestamp = None;
+
+ for hit in &hits {
+ let Some(sort) = hit
+ .get("sort")
+ .and_then(|s| s.as_array())
+ .filter(|a| !a.is_empty())
+ else {
+ warn!(
+ connector_id = self.id,
+ hit_id = hit.get("_id").and_then(|value| value.as_str()),
+ "Skipping OpenSearch hit without sort tuple; document will not be published"
+ );
+ continue;
+ };
+
+ last_sort = Some(sort);
+
+ let Some(source) = hit.get("_source") else {
+ warn!(
+ connector_id = self.id,
+ hit_id = hit.get("_id").and_then(|v| v.as_str()),
+ "Skipping OpenSearch hit without _source; document will not be published"
+ );
+ continue;
+ };
Review Comment:
recent version of opensearch allows disabling _source field even when document contains indexed attributes (making it a valid document): https://docs.opensearch.org/latest/mappings/metadata-fields/source/#disabling-the-field, more so to reduce index size when vector embedding is also stored along. @ryerraguntla Do you think it will affect the functionality here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #3515:
URL: https://github.com/apache/iggy/pull/3515#discussion_r3521554542
##########
core/connectors/sources/opensearch_source/src/lib.rs:
##########
@@ -0,0 +1,1135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::retry::{CircuitBreaker, parse_duration};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use opensearch::{
+ OpenSearch, SearchParts,
+ auth::Credentials,
+ http::{Url, transport::TransportBuilder},
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use serde_json::{Value, json};
+use std::sync::Arc;
+use std::time::Duration;
+use tokio::{sync::Mutex, time::sleep};
+use tracing::{debug, error, info, warn};
+
+mod retry;
+mod state_manager;
+use crate::retry::{
+ DEFAULT_CB_COOL_DOWN, DEFAULT_CB_THRESHOLD, DEFAULT_MAX_OPEN_RETRIES, DEFAULT_MAX_RETRIES,
+ DEFAULT_OPEN_RETRY_MAX_DELAY, DEFAULT_RETRY_DELAY, DEFAULT_RETRY_MAX_DELAY, RetryBackoff,
+ is_transient_status, normalized_max_attempts, sleep_before_retry,
+};
+use crate::state_manager::{SOURCE_STATE_VERSION, SourceState, validate_state_storage_config};
+
+source_connector!(OpenSearchSource);
+
+const CONNECTOR_NAME: &str = "OpenSearch source";
+const DEFAULT_POLLING_INTERVAL: &str = "10s";
+const DEFAULT_BATCH_SIZE: usize = 100;
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct State {
+ #[serde(default)]
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ #[serde(default, alias = "total_documents_fetched")]
+ total_documents_published: usize,
+ #[serde(default)]
+ poll_count: usize,
+ /// OpenSearch `search_after` tuple from the last hit in the previous batch.
+ #[serde(default)]
+ search_after: Option<Vec<Value>>,
+ #[serde(default)]
+ error_count: usize,
+ #[serde(default)]
+ last_error: Option<String>,
+ #[serde(default)]
+ processing_stats: ProcessingStats,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct ProcessingStats {
+ #[serde(default)]
+ total_bytes_processed: u64,
+ /// Running cumulative average over the connector's lifetime, persisted and accumulated
+ /// across restarts. Reflects long-term throughput baseline, not session-only average.
+ #[serde(default)]
+ avg_batch_processing_time_ms: f64,
+ #[serde(default)]
+ last_successful_poll: Option<DateTime<Utc>>,
+ #[serde(default)]
+ empty_polls_count: usize,
+ #[serde(default)]
+ successful_polls_count: usize,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct StateConfig {
+ #[serde(default)]
+ pub enabled: bool,
+ pub storage_type: Option<String>,
+ pub storage_config: Option<Value>,
+ pub state_id: Option<String>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct OpenSearchSourceConfig {
+ pub url: String,
+ pub index: String,
+ pub username: Option<String>,
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_optional_secret")]
+ pub password: Option<SecretString>,
+ pub query: Option<Value>,
+ pub polling_interval: Option<String>,
+ pub batch_size: Option<usize>,
+ pub timestamp_field: Option<String>,
+ #[serde(default)]
+ pub verbose_logging: bool,
+ pub state: Option<StateConfig>,
+ /// Total poll-phase attempt count (not retry count); 1 = no retries, 3 = 2 retries.
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+ pub retry_max_delay: Option<String>,
+ /// Total open-phase attempt count (not retry count); 1 = no retries, 5 = 4 retries.
+ pub max_open_retries: Option<u32>,
+ pub open_retry_max_delay: Option<String>,
+ pub circuit_breaker_threshold: Option<u32>,
+ pub circuit_breaker_cool_down: Option<String>,
+}
+
+#[derive(Debug)]
+pub struct OpenSearchSource {
+ id: u32,
+ config: OpenSearchSourceConfig,
+ client: Option<OpenSearch>,
+ polling_interval: Duration,
+ /// Pre-built query body (query + size + sort). Set once in `open()` after config validation.
+ /// `None` before `open()` and after a failed `open()`.
+ search_body_base: Option<Value>,
+ verbose: bool,
+ max_retries: u32,
+ retry_delay: Duration,
+ retry_max_delay: Duration,
+ max_open_retries: u32,
+ open_retry_max_delay: Duration,
+ circuit_breaker: Arc<CircuitBreaker>,
+ state: Mutex<State>,
+ /// `Some(cause)` when runtime state restore was rejected; `None` means restore succeeded.
+ state_restore_error: Option<String>,
+ /// True when `new()` restored a valid runtime `ConnectorState`. File mirror must not override it.
+ runtime_state_restored: bool,
+}
+
+struct SearchOutcome {
+ messages: Vec<ProducedMessage>,
+ search_after: Option<Vec<Value>>,
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ batch_bytes: u64,
+}
+
+impl OpenSearchSource {
+ pub fn new(id: u32, config: OpenSearchSourceConfig, state: Option<ConnectorState>) -> Self {
+ let polling_interval =
+ parse_duration(config.polling_interval.as_deref(), DEFAULT_POLLING_INTERVAL);
+ let verbose = config.verbose_logging;
+ let (restored_state, state_restore_error, runtime_state_restored) =
+ restore_state(id, state);
+
+ let cb_threshold = config
+ .circuit_breaker_threshold
+ .unwrap_or(DEFAULT_CB_THRESHOLD);
+ let cb_cool_down = parse_duration(
+ config.circuit_breaker_cool_down.as_deref(),
+ DEFAULT_CB_COOL_DOWN,
+ );
+ let circuit_breaker = Arc::new(CircuitBreaker::new(cb_threshold, cb_cool_down));
+ let max_retries =
+ normalized_max_attempts(config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES));
+ let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+ let retry_max_delay =
+ parse_duration(config.retry_max_delay.as_deref(), DEFAULT_RETRY_MAX_DELAY);
+ let max_open_retries =
+ normalized_max_attempts(config.max_open_retries.unwrap_or(DEFAULT_MAX_OPEN_RETRIES));
+ let open_retry_max_delay = parse_duration(
+ config.open_retry_max_delay.as_deref(),
+ DEFAULT_OPEN_RETRY_MAX_DELAY,
+ );
+
+ OpenSearchSource {
+ id,
+ config,
+ max_retries,
+ retry_delay,
+ retry_max_delay,
+ max_open_retries,
+ open_retry_max_delay,
+ circuit_breaker,
+ client: None,
+ polling_interval,
+ search_body_base: None,
+ verbose,
+ state: Mutex::new(restored_state),
+ state_restore_error,
+ runtime_state_restored,
+ }
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn batch_size(&self) -> usize {
+ self.config.batch_size.unwrap_or(DEFAULT_BATCH_SIZE)
+ }
+
+ fn timestamp_field(&self) -> Result<&str, Error> {
+ self.config
+ .timestamp_field
+ .as_deref()
+ .filter(|s| !s.is_empty())
+ .ok_or_else(|| Error::InvalidConfigValue("timestamp_field is required".to_string()))
+ }
+
+ fn get_state_id(&self) -> String {
+ self.config
+ .state
+ .as_ref()
+ .and_then(|s| s.state_id.clone())
+ .unwrap_or_else(|| format!("opensearch_source_{}", self.id))
+ }
+
+ pub(crate) async fn internal_state_to_source_state(&self) -> Result<SourceState, Error> {
+ let state = self.state.lock().await;
+ let data = serde_json::to_value(&*state).map_err(|error| {
+ Error::Serialization(format!("Failed to serialize connector state: {error}"))
+ })?;
+
+ Ok(SourceState {
+ id: self.get_state_id(),
+ last_updated: Utc::now(),
+ version: SOURCE_STATE_VERSION,
+ data,
+ metadata: Some(json!({
+ "connector_type": "opensearch_source",
+ "connector_id": self.id,
+ "index": self.config.index,
+ })),
+ })
+ }
+
+ pub(crate) async fn source_state_to_internal_state(
+ &mut self,
+ source_state: SourceState,
+ ) -> Result<(), Error> {
+ if source_state.version != SOURCE_STATE_VERSION {
+ return Err(Error::Serialization(format!(
+ "unsupported file state version {}, expected {SOURCE_STATE_VERSION}",
+ source_state.version
+ )));
+ }
+
+ let restored: State = serde_json::from_value(source_state.data).map_err(|error| {
+ Error::Serialization(format!("Failed to deserialize connector state: {error}"))
+ })?;
+
+ let mut state = self.state.lock().await;
+ *state = restored;
+ Ok(())
+ }
+
+ async fn create_client(&self) -> Result<OpenSearch, Error> {
+ let url = Url::parse(&self.config.url).map_err(|error| {
+ Error::InvalidConfigValue(format!("Invalid OpenSearch URL: {error}"))
+ })?;
+
+ let conn_pool = opensearch::http::transport::SingleNodeConnectionPool::new(url);
+ let mut transport_builder = TransportBuilder::new(conn_pool);
+
+ if let (Some(username), Some(password)) = (&self.config.username, &self.config.password) {
+ let credentials =
+ Credentials::Basic(username.clone(), password.expose_secret().to_string());
+ transport_builder = transport_builder.auth(credentials);
+ }
+
+ let transport = transport_builder
+ .build()
+ .map_err(|error| Error::InitError(format!("Failed to build transport: {error}")))?;
+
+ Ok(OpenSearch::new(transport))
+ }
+
+ async fn check_index_exists_with_retry(&self, client: &OpenSearch) -> Result<(), Error> {
+ let max_attempts = self.max_open_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let response = match client
+ .indices()
+ .exists(opensearch::indices::IndicesExistsParts::Index(&[&self
+ .config
+ .index]))
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ // retry_delay is shared with the poll phase (no separate open-phase
+ // base delay); open_retry_max_delay caps the growth independently.
+ // Same pattern as the InfluxDB source connector.
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::InitError(format!(
+ "Failed to check index existence: {error}"
+ )));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(());
+ }
+
+ let status = response.status_code().as_u16();
+ if status == 404 {
+ return Err(Error::InitError(format!(
+ "Index '{}' does not exist or is not accessible",
+ self.config.index
+ )));
+ }
+
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::InitError(format!(
+ "Index '{}' check failed: HTTP {status}: {error_text}",
+ self.config.index
+ )));
+ }
+ }
+
+ async fn send_search_with_retry(
+ &self,
+ client: &OpenSearch,
+ mut search_body: Value,
+ ) -> Result<opensearch::http::response::Response, Error> {
+ let max_attempts = self.max_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let body = if attempt < max_attempts {
+ search_body.clone()
+ } else {
+ std::mem::take(&mut search_body)
+ };
+ let response = match client
+ .search(SearchParts::Index(&[&self.config.index]))
+ .body(body)
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::Storage(format!("Failed to execute search: {error}")));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(response);
+ }
+
+ let status = response.status_code().as_u16();
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::Storage(format!(
+ "Search request failed: {error_text}"
+ )));
+ }
+ }
+
+ async fn search_documents(&self, client: &OpenSearch) -> Result<SearchOutcome, Error> {
+ let state = self.state.lock().await;
+ let search_after = state.search_after.clone();
+ drop(state);
+
+ let timestamp_field = self.timestamp_field()?;
+
+ let mut search_body = self
+ .search_body_base
+ .as_ref()
+ .ok_or_else(|| {
+ Error::Connection("connector not initialized; call open() first".to_string())
+ })?
+ .clone();
+
+ if let Some(cursor) = search_after {
+ search_body["search_after"] = json!(cursor);
+ }
+
+ let response = self.send_search_with_retry(client, search_body).await?;
+
+ let mut response_body: Value = response
+ .json()
+ .await
+ .map_err(|e| Error::Storage(format!("Failed to parse search response: {e}")))?;
+
+ let hits: Vec<Value> = response_body
+ .get_mut("hits")
+ .and_then(|h| h.get_mut("hits"))
+ .and_then(|arr| arr.as_array_mut())
+ .map(std::mem::take)
+ .unwrap_or_default();
+
+ let mut messages = Vec::with_capacity(hits.len());
+ let mut batch_bytes = 0u64;
+ let mut last_sort = None;
+ let mut last_poll_timestamp = None;
+
+ for hit in &hits {
+ let Some(sort) = hit
+ .get("sort")
+ .and_then(|s| s.as_array())
+ .filter(|a| !a.is_empty())
+ else {
+ warn!(
+ connector_id = self.id,
+ hit_id = hit.get("_id").and_then(|value| value.as_str()),
+ "Skipping OpenSearch hit without sort tuple; document will not be published"
+ );
+ continue;
+ };
+
+ last_sort = Some(sort);
+
+ let Some(source) = hit.get("_source") else {
+ warn!(
+ connector_id = self.id,
+ hit_id = hit.get("_id").and_then(|v| v.as_str()),
+ "Skipping OpenSearch hit without _source; document will not be published"
+ );
+ continue;
+ };
+
+ if let Some(timestamp_value) = source.get(timestamp_field)
+ && let Some(timestamp_utc) = parse_document_timestamp(timestamp_value)
+ {
+ last_poll_timestamp = Some(timestamp_utc);
+ }
+
+ let payload = serde_json::to_vec(source)
+ .map_err(|e| Error::Serialization(format!("Failed to serialize document: {e}")))?;
+ batch_bytes += payload.len() as u64;
+
+ messages.push(ProducedMessage {
+ id: None,
+ headers: None,
+ checksum: None,
+ timestamp: None,
+ origin_timestamp: None,
+ payload,
+ });
+ }
+
+ if !hits.is_empty() && last_sort.is_none() {
+ return Err(Error::Storage(format!(
+ "OpenSearch returned {} hit(s) but none had a sort tuple; \
+ index may be missing the sort field or using an incompatible mapping",
+ hits.len()
+ )));
+ }
+
+ // Guard: cursor would advance past the entire batch but nothing would be published.
+ // This happens when _source is disabled on the index (all sort-bearing hits lack _source).
+ if !hits.is_empty() && messages.is_empty() && last_sort.is_some() {
+ return Err(Error::Storage(format!(
+ "OpenSearch returned {} hit(s) with valid sort tuples but all were missing \
+ _source; index may have _source disabled. Refusing to advance cursor \
+ without publishing any messages.",
+ hits.len()
+ )));
+ }
+
+ Ok(SearchOutcome {
+ messages,
+ search_after: last_sort.map(ToOwned::to_owned),
+ last_poll_timestamp,
+ batch_bytes,
+ })
+ }
+
+ async fn finalize_poll(
+ &self,
+ outcome: SearchOutcome,
+ processing_time_ms: f64,
+ ) -> (Vec<ProducedMessage>, Option<ConnectorState>) {
+ let mut state = self.state.lock().await;
+ state.total_documents_published += outcome.messages.len();
+ state.poll_count += 1;
+ if let Some(cursor) = outcome.search_after {
+ state.search_after = Some(cursor);
+ }
+ if let Some(timestamp) = outcome.last_poll_timestamp {
+ state.last_poll_timestamp = Some(timestamp);
+ }
+ state.processing_stats.total_bytes_processed += outcome.batch_bytes;
+
+ if outcome.messages.is_empty() {
+ state.processing_stats.empty_polls_count += 1;
+ } else {
+ state.processing_stats.successful_polls_count += 1;
+ state.processing_stats.last_successful_poll = Some(Utc::now());
+ }
+
+ let total_polls = state.processing_stats.successful_polls_count
+ + state.processing_stats.empty_polls_count;
+ // total_polls >= 1 because one of the two counters was just incremented above,
+ // but use saturating_sub to make the invariant machine-checked.
+ state.processing_stats.avg_batch_processing_time_ms =
+ (state.processing_stats.avg_batch_processing_time_ms
+ * total_polls.saturating_sub(1) as f64
+ + processing_time_ms)
+ / total_polls as f64;
+
+ let produced_count = outcome.messages.len();
+ let total_documents_published = state.total_documents_published;
+ let state_snapshot = state.clone();
+ let messages = outcome.messages;
+ drop(state);
+ let persisted_state = self.serialize_state(&state_snapshot);
+
+ if self.verbose {
+ info!(
+ "OpenSearch source connector ID: {} produced {produced_count} messages. \
+ Total published: {total_documents_published}",
+ self.id
+ );
+ } else {
+ debug!(
+ "OpenSearch source connector ID: {} produced {produced_count} messages. \
+ Total published: {total_documents_published}",
+ self.id
+ );
+ }
+
+ (messages, persisted_state)
+ }
+
+ #[cfg(test)]
+ fn client_initialized(&self) -> bool {
+ self.client.is_some()
+ }
+
+ #[cfg(test)]
+ async fn test_metrics(&self) -> (usize, usize, usize, usize) {
+ let state = self.state.lock().await;
+ (
+ state.total_documents_published,
+ state.poll_count,
+ state.error_count,
+ state.processing_stats.empty_polls_count,
+ )
+ }
+
+ #[cfg(test)]
+ async fn test_search_after(&self) -> Option<Vec<Value>> {
+ self.state.lock().await.search_after.clone()
+ }
+
+ #[cfg(test)]
+ async fn test_last_poll_timestamp(&self) -> Option<DateTime<Utc>> {
+ self.state.lock().await.last_poll_timestamp
+ }
+
+ async fn handle_poll_error(&self, error: Error) -> Result<ProducedMessages, Error> {
+ self.circuit_breaker.record_failure().await;
+ let mut state = self.state.lock().await;
+ // error_count and last_error accumulate in State and are captured by the next
+ // finalize_poll call (success path). Both runtime ConnectorState and file state
+ // preserve these values across restarts.
+ state.error_count += 1;
+ state.last_error = Some(error.to_string());
+ drop(state);
+ error!(
+ "{CONNECTOR_NAME} connector ID: {} poll failed: {error}",
+ self.id
+ );
+ Err(error)
+ }
+}
+
+fn restore_state(id: u32, state: Option<ConnectorState>) -> (State, Option<String>, bool) {
+ let Some(connector_state) = state else {
+ return (State::default(), None, false);
+ };
+
+ match connector_state.deserialize::<State>(CONNECTOR_NAME, id) {
+ Some(restored) => {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Documents published: {}, poll count: {}",
+ restored.total_documents_published, restored.poll_count
+ );
+ (restored, None, true)
+ }
+ None => {
+ let cause = "persisted state exists but could not be deserialized. \
+ Refusing to start to prevent silent cursor reset."
+ .to_string();
+ error!("{CONNECTOR_NAME} ID {id}: {cause}");
+ (State::default(), Some(cause), false)
+ }
+ }
+}
+
+fn validate_open_config(config: &OpenSearchSourceConfig) -> Result<(), Error> {
+ if config.timestamp_field.as_deref().is_none_or(str::is_empty) {
+ return Err(Error::InvalidConfigValue(
+ "timestamp_field is required for incremental OpenSearch polling".to_string(),
+ ));
+ }
+
+ if matches!(config.batch_size, Some(0)) {
+ return Err(Error::InvalidConfigValue(
+ "batch_size must be at least 1".to_string(),
+ ));
+ }
+
+ if let Some(state) = &config.state
+ && state.enabled
+ {
+ validate_state_storage_config(state)?;
+ }
+
+ Ok(())
+}
+
+fn parse_document_timestamp(value: &Value) -> Option<DateTime<Utc>> {
+ match value {
+ Value::String(text) => DateTime::parse_from_rfc3339(text)
+ .ok()
+ .map(|timestamp| timestamp.with_timezone(&Utc)),
+ Value::Number(number) => {
+ let raw = number.as_i64()?;
+ // Values above 1e12 are already milliseconds (Unix epoch seconds won't reach
+ // 1e12 until year 33658). Values at or below are treated as seconds and
+ // multiplied by 1000.
+ let millis = if !(-1_000_000_000_000..=1_000_000_000_000).contains(&raw) {
+ raw
+ } else {
+ raw.saturating_mul(1_000)
+ };
+ DateTime::from_timestamp_millis(millis).map(|timestamp| timestamp.with_timezone(&Utc))
+ }
+ _ => None,
+ }
+}
+
+#[async_trait]
+impl Source for OpenSearchSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ if let Some(ref cause) = self.state_restore_error {
+ return Err(Error::InitError(format!("state restore failed: {cause}")));
+ }
+
+ validate_open_config(&self.config)?;
+
+ info!(
+ "Opening OpenSearch source connector with ID: {} for URL: {}, index: {}",
+ self.id, self.config.url, self.config.index
+ );
+
+ let client = self.create_client().await?;
+
+ self.check_index_exists_with_retry(&client).await?;
+
+ let timestamp_field = self.timestamp_field()?;
+ let batch_size = self.batch_size();
+ self.search_body_base = Some(json!({
+ "query": self.config.query.clone().unwrap_or_else(|| json!({ "match_all": {} })),
+ "size": batch_size,
+ "sort": [
+ { timestamp_field: { "order": "asc" } },
+ { "_id": { "order": "asc" } }
+ ]
+ }));
Review Comment:
_id is global unique.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #3515:
URL: https://github.com/apache/iggy/pull/3515#discussion_r3530518819
##########
core/connectors/sources/opensearch_source/src/lib.rs:
##########
@@ -0,0 +1,1135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::retry::{CircuitBreaker, parse_duration};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use opensearch::{
+ OpenSearch, SearchParts,
+ auth::Credentials,
+ http::{Url, transport::TransportBuilder},
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use serde_json::{Value, json};
+use std::sync::Arc;
+use std::time::Duration;
+use tokio::{sync::Mutex, time::sleep};
+use tracing::{debug, error, info, warn};
+
+mod retry;
+mod state_manager;
+use crate::retry::{
+ DEFAULT_CB_COOL_DOWN, DEFAULT_CB_THRESHOLD, DEFAULT_MAX_OPEN_RETRIES, DEFAULT_MAX_RETRIES,
+ DEFAULT_OPEN_RETRY_MAX_DELAY, DEFAULT_RETRY_DELAY, DEFAULT_RETRY_MAX_DELAY, RetryBackoff,
+ is_transient_status, normalized_max_attempts, sleep_before_retry,
+};
+use crate::state_manager::{SOURCE_STATE_VERSION, SourceState, validate_state_storage_config};
+
+source_connector!(OpenSearchSource);
+
+const CONNECTOR_NAME: &str = "OpenSearch source";
+const DEFAULT_POLLING_INTERVAL: &str = "10s";
+const DEFAULT_BATCH_SIZE: usize = 100;
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct State {
+ #[serde(default)]
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ #[serde(default, alias = "total_documents_fetched")]
+ total_documents_published: usize,
+ #[serde(default)]
+ poll_count: usize,
+ /// OpenSearch `search_after` tuple from the last hit in the previous batch.
+ #[serde(default)]
+ search_after: Option<Vec<Value>>,
+ #[serde(default)]
+ error_count: usize,
+ #[serde(default)]
+ last_error: Option<String>,
+ #[serde(default)]
+ processing_stats: ProcessingStats,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct ProcessingStats {
+ #[serde(default)]
+ total_bytes_processed: u64,
+ /// Running cumulative average over the connector's lifetime, persisted and accumulated
+ /// across restarts. Reflects long-term throughput baseline, not session-only average.
+ #[serde(default)]
+ avg_batch_processing_time_ms: f64,
+ #[serde(default)]
+ last_successful_poll: Option<DateTime<Utc>>,
+ #[serde(default)]
+ empty_polls_count: usize,
+ #[serde(default)]
+ successful_polls_count: usize,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct StateConfig {
+ #[serde(default)]
+ pub enabled: bool,
+ pub storage_type: Option<String>,
+ pub storage_config: Option<Value>,
+ pub state_id: Option<String>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct OpenSearchSourceConfig {
+ pub url: String,
+ pub index: String,
+ pub username: Option<String>,
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_optional_secret")]
+ pub password: Option<SecretString>,
+ pub query: Option<Value>,
+ pub polling_interval: Option<String>,
+ pub batch_size: Option<usize>,
+ pub timestamp_field: Option<String>,
+ #[serde(default)]
+ pub verbose_logging: bool,
+ pub state: Option<StateConfig>,
+ /// Total poll-phase attempt count (not retry count); 1 = no retries, 3 = 2 retries.
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+ pub retry_max_delay: Option<String>,
+ /// Total open-phase attempt count (not retry count); 1 = no retries, 5 = 4 retries.
+ pub max_open_retries: Option<u32>,
+ pub open_retry_max_delay: Option<String>,
+ pub circuit_breaker_threshold: Option<u32>,
+ pub circuit_breaker_cool_down: Option<String>,
+}
+
+#[derive(Debug)]
+pub struct OpenSearchSource {
+ id: u32,
+ config: OpenSearchSourceConfig,
+ client: Option<OpenSearch>,
+ polling_interval: Duration,
+ /// Pre-built query body (query + size + sort). Set once in `open()` after config validation.
+ /// `None` before `open()` and after a failed `open()`.
+ search_body_base: Option<Value>,
+ verbose: bool,
+ max_retries: u32,
+ retry_delay: Duration,
+ retry_max_delay: Duration,
+ max_open_retries: u32,
+ open_retry_max_delay: Duration,
+ circuit_breaker: Arc<CircuitBreaker>,
+ state: Mutex<State>,
+ /// `Some(cause)` when runtime state restore was rejected; `None` means restore succeeded.
+ state_restore_error: Option<String>,
+ /// True when `new()` restored a valid runtime `ConnectorState`. File mirror must not override it.
+ runtime_state_restored: bool,
+}
+
+struct SearchOutcome {
+ messages: Vec<ProducedMessage>,
+ search_after: Option<Vec<Value>>,
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ batch_bytes: u64,
+}
+
+impl OpenSearchSource {
+ pub fn new(id: u32, config: OpenSearchSourceConfig, state: Option<ConnectorState>) -> Self {
+ let polling_interval =
+ parse_duration(config.polling_interval.as_deref(), DEFAULT_POLLING_INTERVAL);
+ let verbose = config.verbose_logging;
+ let (restored_state, state_restore_error, runtime_state_restored) =
+ restore_state(id, state);
+
+ let cb_threshold = config
+ .circuit_breaker_threshold
+ .unwrap_or(DEFAULT_CB_THRESHOLD);
+ let cb_cool_down = parse_duration(
+ config.circuit_breaker_cool_down.as_deref(),
+ DEFAULT_CB_COOL_DOWN,
+ );
+ let circuit_breaker = Arc::new(CircuitBreaker::new(cb_threshold, cb_cool_down));
+ let max_retries =
+ normalized_max_attempts(config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES));
+ let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+ let retry_max_delay =
+ parse_duration(config.retry_max_delay.as_deref(), DEFAULT_RETRY_MAX_DELAY);
+ let max_open_retries =
+ normalized_max_attempts(config.max_open_retries.unwrap_or(DEFAULT_MAX_OPEN_RETRIES));
+ let open_retry_max_delay = parse_duration(
+ config.open_retry_max_delay.as_deref(),
+ DEFAULT_OPEN_RETRY_MAX_DELAY,
+ );
+
+ OpenSearchSource {
+ id,
+ config,
+ max_retries,
+ retry_delay,
+ retry_max_delay,
+ max_open_retries,
+ open_retry_max_delay,
+ circuit_breaker,
+ client: None,
+ polling_interval,
+ search_body_base: None,
+ verbose,
+ state: Mutex::new(restored_state),
+ state_restore_error,
+ runtime_state_restored,
+ }
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn batch_size(&self) -> usize {
+ self.config.batch_size.unwrap_or(DEFAULT_BATCH_SIZE)
+ }
+
+ fn timestamp_field(&self) -> Result<&str, Error> {
+ self.config
+ .timestamp_field
+ .as_deref()
+ .filter(|s| !s.is_empty())
+ .ok_or_else(|| Error::InvalidConfigValue("timestamp_field is required".to_string()))
+ }
+
+ fn get_state_id(&self) -> String {
+ self.config
+ .state
+ .as_ref()
+ .and_then(|s| s.state_id.clone())
+ .unwrap_or_else(|| format!("opensearch_source_{}", self.id))
+ }
+
+ pub(crate) async fn internal_state_to_source_state(&self) -> Result<SourceState, Error> {
+ let state = self.state.lock().await;
+ let data = serde_json::to_value(&*state).map_err(|error| {
+ Error::Serialization(format!("Failed to serialize connector state: {error}"))
+ })?;
+
+ Ok(SourceState {
+ id: self.get_state_id(),
+ last_updated: Utc::now(),
+ version: SOURCE_STATE_VERSION,
+ data,
+ metadata: Some(json!({
+ "connector_type": "opensearch_source",
+ "connector_id": self.id,
+ "index": self.config.index,
+ })),
+ })
+ }
+
+ pub(crate) async fn source_state_to_internal_state(
+ &mut self,
+ source_state: SourceState,
+ ) -> Result<(), Error> {
+ if source_state.version != SOURCE_STATE_VERSION {
+ return Err(Error::Serialization(format!(
+ "unsupported file state version {}, expected {SOURCE_STATE_VERSION}",
+ source_state.version
+ )));
+ }
+
+ let restored: State = serde_json::from_value(source_state.data).map_err(|error| {
+ Error::Serialization(format!("Failed to deserialize connector state: {error}"))
+ })?;
+
+ let mut state = self.state.lock().await;
+ *state = restored;
+ Ok(())
+ }
+
+ async fn create_client(&self) -> Result<OpenSearch, Error> {
+ let url = Url::parse(&self.config.url).map_err(|error| {
+ Error::InvalidConfigValue(format!("Invalid OpenSearch URL: {error}"))
+ })?;
+
+ let conn_pool = opensearch::http::transport::SingleNodeConnectionPool::new(url);
+ let mut transport_builder = TransportBuilder::new(conn_pool);
+
+ if let (Some(username), Some(password)) = (&self.config.username, &self.config.password) {
+ let credentials =
+ Credentials::Basic(username.clone(), password.expose_secret().to_string());
+ transport_builder = transport_builder.auth(credentials);
+ }
+
+ let transport = transport_builder
+ .build()
+ .map_err(|error| Error::InitError(format!("Failed to build transport: {error}")))?;
+
+ Ok(OpenSearch::new(transport))
+ }
+
+ async fn check_index_exists_with_retry(&self, client: &OpenSearch) -> Result<(), Error> {
+ let max_attempts = self.max_open_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let response = match client
+ .indices()
+ .exists(opensearch::indices::IndicesExistsParts::Index(&[&self
+ .config
+ .index]))
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ // retry_delay is shared with the poll phase (no separate open-phase
+ // base delay); open_retry_max_delay caps the growth independently.
+ // Same pattern as the InfluxDB source connector.
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::InitError(format!(
+ "Failed to check index existence: {error}"
+ )));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(());
+ }
+
+ let status = response.status_code().as_u16();
+ if status == 404 {
+ return Err(Error::InitError(format!(
+ "Index '{}' does not exist or is not accessible",
+ self.config.index
+ )));
+ }
+
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::InitError(format!(
+ "Index '{}' check failed: HTTP {status}: {error_text}",
+ self.config.index
+ )));
+ }
+ }
+
+ async fn send_search_with_retry(
+ &self,
+ client: &OpenSearch,
+ mut search_body: Value,
+ ) -> Result<opensearch::http::response::Response, Error> {
+ let max_attempts = self.max_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let body = if attempt < max_attempts {
+ search_body.clone()
+ } else {
+ std::mem::take(&mut search_body)
+ };
+ let response = match client
+ .search(SearchParts::Index(&[&self.config.index]))
+ .body(body)
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::Storage(format!("Failed to execute search: {error}")));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(response);
+ }
+
+ let status = response.status_code().as_u16();
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::Storage(format!(
+ "Search request failed: {error_text}"
+ )));
+ }
+ }
+
+ async fn search_documents(&self, client: &OpenSearch) -> Result<SearchOutcome, Error> {
+ let state = self.state.lock().await;
+ let search_after = state.search_after.clone();
+ drop(state);
+
+ let timestamp_field = self.timestamp_field()?;
+
+ let mut search_body = self
+ .search_body_base
+ .as_ref()
+ .ok_or_else(|| {
+ Error::Connection("connector not initialized; call open() first".to_string())
+ })?
+ .clone();
+
+ if let Some(cursor) = search_after {
+ search_body["search_after"] = json!(cursor);
+ }
+
+ let response = self.send_search_with_retry(client, search_body).await?;
+
+ let mut response_body: Value = response
+ .json()
+ .await
+ .map_err(|e| Error::Storage(format!("Failed to parse search response: {e}")))?;
+
+ let hits: Vec<Value> = response_body
+ .get_mut("hits")
+ .and_then(|h| h.get_mut("hits"))
+ .and_then(|arr| arr.as_array_mut())
+ .map(std::mem::take)
+ .unwrap_or_default();
+
+ let mut messages = Vec::with_capacity(hits.len());
+ let mut batch_bytes = 0u64;
+ let mut last_sort = None;
+ let mut last_poll_timestamp = None;
+
+ for hit in &hits {
+ let Some(sort) = hit
+ .get("sort")
+ .and_then(|s| s.as_array())
+ .filter(|a| !a.is_empty())
+ else {
+ warn!(
+ connector_id = self.id,
+ hit_id = hit.get("_id").and_then(|value| value.as_str()),
+ "Skipping OpenSearch hit without sort tuple; document will not be published"
+ );
+ continue;
+ };
+
+ last_sort = Some(sort);
Review Comment:
how would the test case help - I am habituated with writing the use cases, test cases then start writing the code - TDD. Thinking of different data set scenarios and mix of data is one of the criteria in the Igyy connectors development. By writing test cases with mix of data so that cursor handles the bad data and continues with the expected behavior. This is the reason, my source contributions have a higher percentage of code coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #3515:
URL: https://github.com/apache/iggy/pull/3515#discussion_r3509956095
##########
core/connectors/runtime/example_config/connectors/opensearch_source.toml:
##########
@@ -0,0 +1,63 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+type = "source"
+key = "opensearch"
+enabled = true
+version = 0
+name = "OpenSearch source"
+path = "<BASE_DIR>/target/release/libiggy_connector_opensearch_source"
+plugin_config_format = "json"
+
+[[streams]]
+stream = "opensearch_stream"
+topic = "documents"
+schema = "json"
+batch_length = 100
+linger_time = "5ms"
+
+[plugin_config]
+url = "http://localhost:9200"
+index = "logs-*"
+polling_interval = "10s"
+batch_size = 100
+timestamp_field = "@timestamp"
+# username = "admin"
+# password = "replace_with_secret"
+# verbose_logging = false
+
+# HTTP resilience (defaults shown; see docs/RESILIENCE.md).
Review Comment:
It should mention the opensearch projects repo name as well. Removing as suggested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #3515:
URL: https://github.com/apache/iggy/pull/3515#issuecomment-4755934272
## [Codecov](https://app.codecov.io/gh/apache/iggy/pull/3515?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 73.71%. Comparing base ([`bbc03e7`](https://app.codecov.io/gh/apache/iggy/commit/bbc03e7a93760a35359514bcdddc1aadf4587b19?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dd96d1e`](https://app.codecov.io/gh/apache/iggy/commit/dd96d1e36d062b705cfeb54e53531e1611a5d334?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 51 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #3515 +/- ##
============================================
- Coverage 74.40% 73.71% -0.70%
+ Complexity 943 937 -6
============================================
Files 1236 1189 -47
Lines 120862 117286 -3576
Branches 97599 97603 +4
============================================
- Hits 89933 86453 -3480
+ Misses 27972 27821 -151
- Partials 2957 3012 +55
```
| [Components](https://app.codecov.io/gh/apache/iggy/pull/3515/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [Rust Core](https://app.codecov.io/gh/apache/iggy/pull/3515/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.54% <ø> (ø)` | |
| [Java SDK](https://app.codecov.io/gh/apache/iggy/pull/3515/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `58.57% <ø> (+0.13%)` | :arrow_up: |
| [C# SDK](https://app.codecov.io/gh/apache/iggy/pull/3515/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `71.40% <ø> (+1.47%)` | :arrow_up: |
| [Python SDK](https://app.codecov.io/gh/apache/iggy/pull/3515/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `88.88% <ø> (+7.82%)` | :arrow_up: |
| [Node SDK](https://app.codecov.io/gh/apache/iggy/pull/3515/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `87.27% <ø> (-4.27%)` | :arrow_down: |
| [Go SDK](https://app.codecov.io/gh/apache/iggy/pull/3515/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `40.36% <ø> (+0.16%)` | :arrow_up: |
[see 160 files with indirect coverage changes](https://app.codecov.io/gh/apache/iggy/pull/3515/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feat(connectors)/opensearch source connector [iggy]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3515:
URL: https://github.com/apache/iggy/pull/3515#issuecomment-4755928771
<!-- iggy-pr-triage-welcome -->
Thanks for the PR. It is labeled `S-waiting-on-review` and queued for review.
Slash commands (own line, regular comment) move it around the queue:
- `/ready` - back to `S-waiting-on-review` after addressing feedback
- `/author` - flip to `S-waiting-on-author` while you finish changes
- `/request-review @user-or-team` - request a reviewer
See [CONTRIBUTING.md](https://github.com/apache/iggy/blob/master/CONTRIBUTING.md#pr-triage-commands) for details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on PR #3515:
URL: https://github.com/apache/iggy/pull/3515#issuecomment-4756448671
/ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "kriti-sc (via GitHub)" <gi...@apache.org>.
kriti-sc commented on code in PR #3515:
URL: https://github.com/apache/iggy/pull/3515#discussion_r3525266612
##########
core/connectors/sources/opensearch_source/src/lib.rs:
##########
@@ -0,0 +1,1135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::retry::{CircuitBreaker, parse_duration};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use opensearch::{
+ OpenSearch, SearchParts,
+ auth::Credentials,
+ http::{Url, transport::TransportBuilder},
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use serde_json::{Value, json};
+use std::sync::Arc;
+use std::time::Duration;
+use tokio::{sync::Mutex, time::sleep};
+use tracing::{debug, error, info, warn};
+
+mod retry;
+mod state_manager;
+use crate::retry::{
+ DEFAULT_CB_COOL_DOWN, DEFAULT_CB_THRESHOLD, DEFAULT_MAX_OPEN_RETRIES, DEFAULT_MAX_RETRIES,
+ DEFAULT_OPEN_RETRY_MAX_DELAY, DEFAULT_RETRY_DELAY, DEFAULT_RETRY_MAX_DELAY, RetryBackoff,
+ is_transient_status, normalized_max_attempts, sleep_before_retry,
+};
+use crate::state_manager::{SOURCE_STATE_VERSION, SourceState, validate_state_storage_config};
+
+source_connector!(OpenSearchSource);
+
+const CONNECTOR_NAME: &str = "OpenSearch source";
+const DEFAULT_POLLING_INTERVAL: &str = "10s";
+const DEFAULT_BATCH_SIZE: usize = 100;
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct State {
+ #[serde(default)]
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ #[serde(default, alias = "total_documents_fetched")]
+ total_documents_published: usize,
+ #[serde(default)]
+ poll_count: usize,
+ /// OpenSearch `search_after` tuple from the last hit in the previous batch.
+ #[serde(default)]
+ search_after: Option<Vec<Value>>,
+ #[serde(default)]
+ error_count: usize,
+ #[serde(default)]
+ last_error: Option<String>,
+ #[serde(default)]
+ processing_stats: ProcessingStats,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct ProcessingStats {
+ #[serde(default)]
+ total_bytes_processed: u64,
+ /// Running cumulative average over the connector's lifetime, persisted and accumulated
+ /// across restarts. Reflects long-term throughput baseline, not session-only average.
+ #[serde(default)]
+ avg_batch_processing_time_ms: f64,
+ #[serde(default)]
+ last_successful_poll: Option<DateTime<Utc>>,
+ #[serde(default)]
+ empty_polls_count: usize,
+ #[serde(default)]
+ successful_polls_count: usize,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct StateConfig {
+ #[serde(default)]
+ pub enabled: bool,
+ pub storage_type: Option<String>,
+ pub storage_config: Option<Value>,
+ pub state_id: Option<String>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct OpenSearchSourceConfig {
+ pub url: String,
+ pub index: String,
+ pub username: Option<String>,
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_optional_secret")]
+ pub password: Option<SecretString>,
+ pub query: Option<Value>,
+ pub polling_interval: Option<String>,
+ pub batch_size: Option<usize>,
+ pub timestamp_field: Option<String>,
+ #[serde(default)]
+ pub verbose_logging: bool,
+ pub state: Option<StateConfig>,
+ /// Total poll-phase attempt count (not retry count); 1 = no retries, 3 = 2 retries.
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+ pub retry_max_delay: Option<String>,
+ /// Total open-phase attempt count (not retry count); 1 = no retries, 5 = 4 retries.
+ pub max_open_retries: Option<u32>,
+ pub open_retry_max_delay: Option<String>,
+ pub circuit_breaker_threshold: Option<u32>,
+ pub circuit_breaker_cool_down: Option<String>,
+}
+
+#[derive(Debug)]
+pub struct OpenSearchSource {
+ id: u32,
+ config: OpenSearchSourceConfig,
+ client: Option<OpenSearch>,
+ polling_interval: Duration,
+ /// Pre-built query body (query + size + sort). Set once in `open()` after config validation.
+ /// `None` before `open()` and after a failed `open()`.
+ search_body_base: Option<Value>,
+ verbose: bool,
+ max_retries: u32,
+ retry_delay: Duration,
+ retry_max_delay: Duration,
+ max_open_retries: u32,
+ open_retry_max_delay: Duration,
+ circuit_breaker: Arc<CircuitBreaker>,
+ state: Mutex<State>,
+ /// `Some(cause)` when runtime state restore was rejected; `None` means restore succeeded.
+ state_restore_error: Option<String>,
+ /// True when `new()` restored a valid runtime `ConnectorState`. File mirror must not override it.
+ runtime_state_restored: bool,
+}
+
+struct SearchOutcome {
+ messages: Vec<ProducedMessage>,
+ search_after: Option<Vec<Value>>,
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ batch_bytes: u64,
+}
+
+impl OpenSearchSource {
+ pub fn new(id: u32, config: OpenSearchSourceConfig, state: Option<ConnectorState>) -> Self {
+ let polling_interval =
+ parse_duration(config.polling_interval.as_deref(), DEFAULT_POLLING_INTERVAL);
+ let verbose = config.verbose_logging;
+ let (restored_state, state_restore_error, runtime_state_restored) =
+ restore_state(id, state);
+
+ let cb_threshold = config
+ .circuit_breaker_threshold
+ .unwrap_or(DEFAULT_CB_THRESHOLD);
+ let cb_cool_down = parse_duration(
+ config.circuit_breaker_cool_down.as_deref(),
+ DEFAULT_CB_COOL_DOWN,
+ );
+ let circuit_breaker = Arc::new(CircuitBreaker::new(cb_threshold, cb_cool_down));
+ let max_retries =
+ normalized_max_attempts(config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES));
+ let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+ let retry_max_delay =
+ parse_duration(config.retry_max_delay.as_deref(), DEFAULT_RETRY_MAX_DELAY);
+ let max_open_retries =
+ normalized_max_attempts(config.max_open_retries.unwrap_or(DEFAULT_MAX_OPEN_RETRIES));
+ let open_retry_max_delay = parse_duration(
+ config.open_retry_max_delay.as_deref(),
+ DEFAULT_OPEN_RETRY_MAX_DELAY,
+ );
+
+ OpenSearchSource {
+ id,
+ config,
+ max_retries,
+ retry_delay,
+ retry_max_delay,
+ max_open_retries,
+ open_retry_max_delay,
+ circuit_breaker,
+ client: None,
+ polling_interval,
+ search_body_base: None,
+ verbose,
+ state: Mutex::new(restored_state),
+ state_restore_error,
+ runtime_state_restored,
+ }
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn batch_size(&self) -> usize {
+ self.config.batch_size.unwrap_or(DEFAULT_BATCH_SIZE)
+ }
+
+ fn timestamp_field(&self) -> Result<&str, Error> {
+ self.config
+ .timestamp_field
+ .as_deref()
+ .filter(|s| !s.is_empty())
+ .ok_or_else(|| Error::InvalidConfigValue("timestamp_field is required".to_string()))
+ }
+
+ fn get_state_id(&self) -> String {
+ self.config
+ .state
+ .as_ref()
+ .and_then(|s| s.state_id.clone())
+ .unwrap_or_else(|| format!("opensearch_source_{}", self.id))
+ }
+
+ pub(crate) async fn internal_state_to_source_state(&self) -> Result<SourceState, Error> {
+ let state = self.state.lock().await;
+ let data = serde_json::to_value(&*state).map_err(|error| {
+ Error::Serialization(format!("Failed to serialize connector state: {error}"))
+ })?;
+
+ Ok(SourceState {
+ id: self.get_state_id(),
+ last_updated: Utc::now(),
+ version: SOURCE_STATE_VERSION,
+ data,
+ metadata: Some(json!({
+ "connector_type": "opensearch_source",
+ "connector_id": self.id,
+ "index": self.config.index,
+ })),
+ })
+ }
+
+ pub(crate) async fn source_state_to_internal_state(
+ &mut self,
+ source_state: SourceState,
+ ) -> Result<(), Error> {
+ if source_state.version != SOURCE_STATE_VERSION {
+ return Err(Error::Serialization(format!(
+ "unsupported file state version {}, expected {SOURCE_STATE_VERSION}",
+ source_state.version
+ )));
+ }
+
+ let restored: State = serde_json::from_value(source_state.data).map_err(|error| {
+ Error::Serialization(format!("Failed to deserialize connector state: {error}"))
+ })?;
+
+ let mut state = self.state.lock().await;
+ *state = restored;
+ Ok(())
+ }
+
+ async fn create_client(&self) -> Result<OpenSearch, Error> {
+ let url = Url::parse(&self.config.url).map_err(|error| {
+ Error::InvalidConfigValue(format!("Invalid OpenSearch URL: {error}"))
+ })?;
+
+ let conn_pool = opensearch::http::transport::SingleNodeConnectionPool::new(url);
+ let mut transport_builder = TransportBuilder::new(conn_pool);
+
+ if let (Some(username), Some(password)) = (&self.config.username, &self.config.password) {
+ let credentials =
+ Credentials::Basic(username.clone(), password.expose_secret().to_string());
+ transport_builder = transport_builder.auth(credentials);
+ }
+
+ let transport = transport_builder
+ .build()
+ .map_err(|error| Error::InitError(format!("Failed to build transport: {error}")))?;
+
+ Ok(OpenSearch::new(transport))
+ }
+
+ async fn check_index_exists_with_retry(&self, client: &OpenSearch) -> Result<(), Error> {
+ let max_attempts = self.max_open_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let response = match client
+ .indices()
+ .exists(opensearch::indices::IndicesExistsParts::Index(&[&self
+ .config
+ .index]))
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ // retry_delay is shared with the poll phase (no separate open-phase
+ // base delay); open_retry_max_delay caps the growth independently.
+ // Same pattern as the InfluxDB source connector.
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::InitError(format!(
+ "Failed to check index existence: {error}"
+ )));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(());
+ }
+
+ let status = response.status_code().as_u16();
+ if status == 404 {
+ return Err(Error::InitError(format!(
+ "Index '{}' does not exist or is not accessible",
+ self.config.index
+ )));
+ }
+
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::InitError(format!(
+ "Index '{}' check failed: HTTP {status}: {error_text}",
+ self.config.index
+ )));
+ }
+ }
+
+ async fn send_search_with_retry(
+ &self,
+ client: &OpenSearch,
+ mut search_body: Value,
+ ) -> Result<opensearch::http::response::Response, Error> {
+ let max_attempts = self.max_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let body = if attempt < max_attempts {
+ search_body.clone()
+ } else {
+ std::mem::take(&mut search_body)
+ };
+ let response = match client
+ .search(SearchParts::Index(&[&self.config.index]))
+ .body(body)
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::Storage(format!("Failed to execute search: {error}")));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(response);
+ }
+
+ let status = response.status_code().as_u16();
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::Storage(format!(
+ "Search request failed: {error_text}"
+ )));
+ }
+ }
+
+ async fn search_documents(&self, client: &OpenSearch) -> Result<SearchOutcome, Error> {
+ let state = self.state.lock().await;
+ let search_after = state.search_after.clone();
+ drop(state);
+
+ let timestamp_field = self.timestamp_field()?;
+
+ let mut search_body = self
+ .search_body_base
+ .as_ref()
+ .ok_or_else(|| {
+ Error::Connection("connector not initialized; call open() first".to_string())
+ })?
+ .clone();
+
+ if let Some(cursor) = search_after {
+ search_body["search_after"] = json!(cursor);
+ }
+
+ let response = self.send_search_with_retry(client, search_body).await?;
+
+ let mut response_body: Value = response
+ .json()
+ .await
+ .map_err(|e| Error::Storage(format!("Failed to parse search response: {e}")))?;
+
+ let hits: Vec<Value> = response_body
+ .get_mut("hits")
+ .and_then(|h| h.get_mut("hits"))
+ .and_then(|arr| arr.as_array_mut())
+ .map(std::mem::take)
+ .unwrap_or_default();
+
+ let mut messages = Vec::with_capacity(hits.len());
+ let mut batch_bytes = 0u64;
+ let mut last_sort = None;
+ let mut last_poll_timestamp = None;
+
+ for hit in &hits {
+ let Some(sort) = hit
+ .get("sort")
+ .and_then(|s| s.as_array())
+ .filter(|a| !a.is_empty())
+ else {
+ warn!(
+ connector_id = self.id,
+ hit_id = hit.get("_id").and_then(|value| value.as_str()),
+ "Skipping OpenSearch hit without sort tuple; document will not be published"
+ );
+ continue;
+ };
+
+ last_sort = Some(sort);
Review Comment:
i am not sure if i understand -- how would the test case help?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "kriti-sc (via GitHub)" <gi...@apache.org>.
kriti-sc commented on code in PR #3515:
URL: https://github.com/apache/iggy/pull/3515#discussion_r3521157160
##########
core/connectors/sources/opensearch_source/src/state_manager.rs:
##########
@@ -0,0 +1,314 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::{OpenSearchSource, StateConfig};
+use async_trait::async_trait;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::Error;
+use serde::{Deserialize, Serialize};
+use std::io::ErrorKind;
+use std::sync::Arc;
+use tracing::info;
+
+pub(crate) const SOURCE_STATE_VERSION: u32 = 1;
+
+impl OpenSearchSource {
+ pub(super) async fn save_state(&self) -> Result<(), Error> {
+ let Some(state_config) = self.config.state.as_ref().filter(|s| s.enabled) else {
+ return Ok(());
+ };
+ let storage = create_state_storage(state_config)?;
+
+ let source_state = self.internal_state_to_source_state().await?;
+ storage.save_source_state(&source_state).await?;
+
+ info!(
+ "Saved state for OpenSearch source connector with ID: {}",
+ self.id
+ );
+ Ok(())
+ }
+
+ pub(super) async fn load_state(&mut self) -> Result<(), Error> {
+ let Some(state_config) = self.config.state.as_ref().filter(|s| s.enabled) else {
+ return Ok(());
+ };
+ let storage = create_state_storage(state_config)?;
+
+ let state_id = self.get_state_id();
+ if let Some(source_state) = storage.load_source_state(&state_id).await? {
+ self.source_state_to_internal_state(source_state).await?;
+
+ let (last_poll_timestamp, total_documents_published, poll_count) = {
+ let state = self.state.lock().await;
+ (
+ state.last_poll_timestamp,
+ state.total_documents_published,
+ state.poll_count,
+ )
+ };
+ info!(
+ "Loaded state for OpenSearch source connector with ID: {} - last poll: {:?}, total docs: {}, polls: {}",
+ self.id, last_poll_timestamp, total_documents_published, poll_count
+ );
+ } else {
+ info!(
+ "No existing state found for OpenSearch source connector with ID: {}, starting fresh",
+ self.id
+ );
+ }
+
+ Ok(())
+ }
+}
+
+pub(crate) fn validate_state_storage_config(config: &StateConfig) -> Result<(), Error> {
+ match config.storage_type.as_deref() {
+ Some("file") | None => Ok(()),
+ Some(storage_type) => Err(Error::InvalidConfigValue(format!(
+ "state storage_type {storage_type:?} is not supported; only \"file\" is implemented"
+ ))),
+ }
+}
+
+pub(crate) fn create_state_storage(config: &StateConfig) -> Result<Arc<dyn StateStorage>, Error> {
+ let base_path = config
+ .storage_config
+ .as_ref()
+ .and_then(|c| c.get("base_path"))
+ .and_then(|p| p.as_str())
+ .unwrap_or("./connector_states");
+
+ Ok(Arc::new(FileStateStorage::new(base_path)))
+}
+
+/// Optional file-backed mirror of connector state. Runtime `ConnectorState` msgpack is authoritative.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub(crate) struct SourceState {
+ pub id: String,
+ pub last_updated: DateTime<Utc>,
+ #[serde(default)]
+ pub version: u32,
+ pub data: serde_json::Value,
+ pub metadata: Option<serde_json::Value>,
+}
+
+#[async_trait]
+pub(crate) trait StateStorage: Send + Sync {
+ async fn save_source_state(&self, state: &SourceState) -> Result<(), Error>;
+
+ async fn load_source_state(&self, id: &str) -> Result<Option<SourceState>, Error>;
+}
+
+pub(crate) struct FileStateStorage {
+ base_path: std::path::PathBuf,
+}
+
+impl FileStateStorage {
+ pub(crate) fn new<P: AsRef<std::path::Path>>(base_path: P) -> Self {
+ Self {
+ base_path: base_path.as_ref().to_path_buf(),
+ }
+ }
+
+ fn get_state_path(&self, id: &str) -> std::path::PathBuf {
+ self.base_path.join(format!("{id}.json"))
+ }
+}
+
+#[async_trait]
+impl StateStorage for FileStateStorage {
+ async fn save_source_state(&self, state: &SourceState) -> Result<(), Error> {
+ use tokio::fs::{self, OpenOptions};
+ use tokio::io::AsyncWriteExt;
+
+ fs::create_dir_all(&self.base_path)
+ .await
+ .map_err(|e| Error::Storage(format!("Failed to create state directory: {e}")))?;
+
+ let path = self.get_state_path(&state.id);
+ // PID in the suffix avoids races when two processes write the same state_id.
+ let tmp_path = path.with_extension(format!("json.{}.tmp", std::process::id()));
+ let json = serde_json::to_string(state)
+ .map_err(|e| Error::Serialization(format!("Failed to serialize source state: {e}")))?;
+
+ let mut tmp_file = OpenOptions::new()
+ .write(true)
+ .create(true)
+ .truncate(true)
+ .open(&tmp_path)
+ .await
+ .map_err(|e| Error::Storage(format!("Failed to open state temp file: {e}")))?;
+ if let Err(e) = tmp_file.write_all(json.as_bytes()).await {
+ let _ = fs::remove_file(&tmp_path).await;
+ return Err(Error::Storage(format!(
+ "Failed to write state temp file: {e}"
+ )));
+ }
Review Comment:
should the file being written be given any permissions here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "kriti-sc (via GitHub)" <gi...@apache.org>.
kriti-sc commented on code in PR #3515:
URL: https://github.com/apache/iggy/pull/3515#discussion_r3521237373
##########
core/connectors/sources/opensearch_source/src/lib.rs:
##########
@@ -0,0 +1,1135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::retry::{CircuitBreaker, parse_duration};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use opensearch::{
+ OpenSearch, SearchParts,
+ auth::Credentials,
+ http::{Url, transport::TransportBuilder},
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use serde_json::{Value, json};
+use std::sync::Arc;
+use std::time::Duration;
+use tokio::{sync::Mutex, time::sleep};
+use tracing::{debug, error, info, warn};
+
+mod retry;
+mod state_manager;
+use crate::retry::{
+ DEFAULT_CB_COOL_DOWN, DEFAULT_CB_THRESHOLD, DEFAULT_MAX_OPEN_RETRIES, DEFAULT_MAX_RETRIES,
+ DEFAULT_OPEN_RETRY_MAX_DELAY, DEFAULT_RETRY_DELAY, DEFAULT_RETRY_MAX_DELAY, RetryBackoff,
+ is_transient_status, normalized_max_attempts, sleep_before_retry,
+};
+use crate::state_manager::{SOURCE_STATE_VERSION, SourceState, validate_state_storage_config};
+
+source_connector!(OpenSearchSource);
+
+const CONNECTOR_NAME: &str = "OpenSearch source";
+const DEFAULT_POLLING_INTERVAL: &str = "10s";
+const DEFAULT_BATCH_SIZE: usize = 100;
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct State {
+ #[serde(default)]
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ #[serde(default, alias = "total_documents_fetched")]
+ total_documents_published: usize,
+ #[serde(default)]
+ poll_count: usize,
+ /// OpenSearch `search_after` tuple from the last hit in the previous batch.
+ #[serde(default)]
+ search_after: Option<Vec<Value>>,
+ #[serde(default)]
+ error_count: usize,
+ #[serde(default)]
+ last_error: Option<String>,
+ #[serde(default)]
+ processing_stats: ProcessingStats,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct ProcessingStats {
+ #[serde(default)]
+ total_bytes_processed: u64,
+ /// Running cumulative average over the connector's lifetime, persisted and accumulated
+ /// across restarts. Reflects long-term throughput baseline, not session-only average.
+ #[serde(default)]
+ avg_batch_processing_time_ms: f64,
+ #[serde(default)]
+ last_successful_poll: Option<DateTime<Utc>>,
+ #[serde(default)]
+ empty_polls_count: usize,
+ #[serde(default)]
+ successful_polls_count: usize,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct StateConfig {
+ #[serde(default)]
+ pub enabled: bool,
+ pub storage_type: Option<String>,
+ pub storage_config: Option<Value>,
+ pub state_id: Option<String>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct OpenSearchSourceConfig {
+ pub url: String,
+ pub index: String,
+ pub username: Option<String>,
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_optional_secret")]
+ pub password: Option<SecretString>,
+ pub query: Option<Value>,
+ pub polling_interval: Option<String>,
+ pub batch_size: Option<usize>,
+ pub timestamp_field: Option<String>,
+ #[serde(default)]
+ pub verbose_logging: bool,
+ pub state: Option<StateConfig>,
+ /// Total poll-phase attempt count (not retry count); 1 = no retries, 3 = 2 retries.
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+ pub retry_max_delay: Option<String>,
+ /// Total open-phase attempt count (not retry count); 1 = no retries, 5 = 4 retries.
+ pub max_open_retries: Option<u32>,
+ pub open_retry_max_delay: Option<String>,
+ pub circuit_breaker_threshold: Option<u32>,
+ pub circuit_breaker_cool_down: Option<String>,
+}
+
+#[derive(Debug)]
+pub struct OpenSearchSource {
+ id: u32,
+ config: OpenSearchSourceConfig,
+ client: Option<OpenSearch>,
+ polling_interval: Duration,
+ /// Pre-built query body (query + size + sort). Set once in `open()` after config validation.
+ /// `None` before `open()` and after a failed `open()`.
+ search_body_base: Option<Value>,
+ verbose: bool,
+ max_retries: u32,
+ retry_delay: Duration,
+ retry_max_delay: Duration,
+ max_open_retries: u32,
+ open_retry_max_delay: Duration,
+ circuit_breaker: Arc<CircuitBreaker>,
+ state: Mutex<State>,
+ /// `Some(cause)` when runtime state restore was rejected; `None` means restore succeeded.
+ state_restore_error: Option<String>,
+ /// True when `new()` restored a valid runtime `ConnectorState`. File mirror must not override it.
+ runtime_state_restored: bool,
+}
+
+struct SearchOutcome {
+ messages: Vec<ProducedMessage>,
+ search_after: Option<Vec<Value>>,
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ batch_bytes: u64,
+}
+
+impl OpenSearchSource {
+ pub fn new(id: u32, config: OpenSearchSourceConfig, state: Option<ConnectorState>) -> Self {
+ let polling_interval =
+ parse_duration(config.polling_interval.as_deref(), DEFAULT_POLLING_INTERVAL);
+ let verbose = config.verbose_logging;
+ let (restored_state, state_restore_error, runtime_state_restored) =
+ restore_state(id, state);
+
+ let cb_threshold = config
+ .circuit_breaker_threshold
+ .unwrap_or(DEFAULT_CB_THRESHOLD);
+ let cb_cool_down = parse_duration(
+ config.circuit_breaker_cool_down.as_deref(),
+ DEFAULT_CB_COOL_DOWN,
+ );
+ let circuit_breaker = Arc::new(CircuitBreaker::new(cb_threshold, cb_cool_down));
+ let max_retries =
+ normalized_max_attempts(config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES));
+ let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+ let retry_max_delay =
+ parse_duration(config.retry_max_delay.as_deref(), DEFAULT_RETRY_MAX_DELAY);
+ let max_open_retries =
+ normalized_max_attempts(config.max_open_retries.unwrap_or(DEFAULT_MAX_OPEN_RETRIES));
+ let open_retry_max_delay = parse_duration(
+ config.open_retry_max_delay.as_deref(),
+ DEFAULT_OPEN_RETRY_MAX_DELAY,
+ );
+
+ OpenSearchSource {
+ id,
+ config,
+ max_retries,
+ retry_delay,
+ retry_max_delay,
+ max_open_retries,
+ open_retry_max_delay,
+ circuit_breaker,
+ client: None,
+ polling_interval,
+ search_body_base: None,
+ verbose,
+ state: Mutex::new(restored_state),
+ state_restore_error,
+ runtime_state_restored,
+ }
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn batch_size(&self) -> usize {
+ self.config.batch_size.unwrap_or(DEFAULT_BATCH_SIZE)
+ }
+
+ fn timestamp_field(&self) -> Result<&str, Error> {
+ self.config
+ .timestamp_field
+ .as_deref()
+ .filter(|s| !s.is_empty())
+ .ok_or_else(|| Error::InvalidConfigValue("timestamp_field is required".to_string()))
+ }
+
+ fn get_state_id(&self) -> String {
+ self.config
+ .state
+ .as_ref()
+ .and_then(|s| s.state_id.clone())
+ .unwrap_or_else(|| format!("opensearch_source_{}", self.id))
+ }
+
+ pub(crate) async fn internal_state_to_source_state(&self) -> Result<SourceState, Error> {
+ let state = self.state.lock().await;
+ let data = serde_json::to_value(&*state).map_err(|error| {
+ Error::Serialization(format!("Failed to serialize connector state: {error}"))
+ })?;
+
+ Ok(SourceState {
+ id: self.get_state_id(),
+ last_updated: Utc::now(),
+ version: SOURCE_STATE_VERSION,
+ data,
+ metadata: Some(json!({
+ "connector_type": "opensearch_source",
+ "connector_id": self.id,
+ "index": self.config.index,
+ })),
+ })
+ }
+
+ pub(crate) async fn source_state_to_internal_state(
+ &mut self,
+ source_state: SourceState,
+ ) -> Result<(), Error> {
+ if source_state.version != SOURCE_STATE_VERSION {
+ return Err(Error::Serialization(format!(
+ "unsupported file state version {}, expected {SOURCE_STATE_VERSION}",
+ source_state.version
+ )));
+ }
+
+ let restored: State = serde_json::from_value(source_state.data).map_err(|error| {
+ Error::Serialization(format!("Failed to deserialize connector state: {error}"))
+ })?;
+
+ let mut state = self.state.lock().await;
+ *state = restored;
+ Ok(())
+ }
+
+ async fn create_client(&self) -> Result<OpenSearch, Error> {
+ let url = Url::parse(&self.config.url).map_err(|error| {
+ Error::InvalidConfigValue(format!("Invalid OpenSearch URL: {error}"))
+ })?;
+
+ let conn_pool = opensearch::http::transport::SingleNodeConnectionPool::new(url);
+ let mut transport_builder = TransportBuilder::new(conn_pool);
+
+ if let (Some(username), Some(password)) = (&self.config.username, &self.config.password) {
+ let credentials =
+ Credentials::Basic(username.clone(), password.expose_secret().to_string());
+ transport_builder = transport_builder.auth(credentials);
+ }
+
+ let transport = transport_builder
+ .build()
+ .map_err(|error| Error::InitError(format!("Failed to build transport: {error}")))?;
+
+ Ok(OpenSearch::new(transport))
+ }
+
+ async fn check_index_exists_with_retry(&self, client: &OpenSearch) -> Result<(), Error> {
+ let max_attempts = self.max_open_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let response = match client
+ .indices()
+ .exists(opensearch::indices::IndicesExistsParts::Index(&[&self
+ .config
+ .index]))
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ // retry_delay is shared with the poll phase (no separate open-phase
+ // base delay); open_retry_max_delay caps the growth independently.
+ // Same pattern as the InfluxDB source connector.
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::InitError(format!(
+ "Failed to check index existence: {error}"
+ )));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(());
+ }
+
+ let status = response.status_code().as_u16();
+ if status == 404 {
+ return Err(Error::InitError(format!(
+ "Index '{}' does not exist or is not accessible",
+ self.config.index
+ )));
+ }
+
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::InitError(format!(
+ "Index '{}' check failed: HTTP {status}: {error_text}",
+ self.config.index
+ )));
+ }
+ }
+
+ async fn send_search_with_retry(
+ &self,
+ client: &OpenSearch,
+ mut search_body: Value,
+ ) -> Result<opensearch::http::response::Response, Error> {
+ let max_attempts = self.max_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let body = if attempt < max_attempts {
+ search_body.clone()
+ } else {
+ std::mem::take(&mut search_body)
+ };
+ let response = match client
+ .search(SearchParts::Index(&[&self.config.index]))
+ .body(body)
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::Storage(format!("Failed to execute search: {error}")));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(response);
+ }
+
+ let status = response.status_code().as_u16();
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::Storage(format!(
+ "Search request failed: {error_text}"
+ )));
+ }
+ }
+
+ async fn search_documents(&self, client: &OpenSearch) -> Result<SearchOutcome, Error> {
+ let state = self.state.lock().await;
+ let search_after = state.search_after.clone();
+ drop(state);
+
+ let timestamp_field = self.timestamp_field()?;
+
+ let mut search_body = self
+ .search_body_base
+ .as_ref()
+ .ok_or_else(|| {
+ Error::Connection("connector not initialized; call open() first".to_string())
+ })?
+ .clone();
+
+ if let Some(cursor) = search_after {
+ search_body["search_after"] = json!(cursor);
+ }
+
+ let response = self.send_search_with_retry(client, search_body).await?;
+
+ let mut response_body: Value = response
+ .json()
+ .await
+ .map_err(|e| Error::Storage(format!("Failed to parse search response: {e}")))?;
+
+ let hits: Vec<Value> = response_body
+ .get_mut("hits")
+ .and_then(|h| h.get_mut("hits"))
+ .and_then(|arr| arr.as_array_mut())
+ .map(std::mem::take)
+ .unwrap_or_default();
+
+ let mut messages = Vec::with_capacity(hits.len());
+ let mut batch_bytes = 0u64;
+ let mut last_sort = None;
+ let mut last_poll_timestamp = None;
+
+ for hit in &hits {
+ let Some(sort) = hit
+ .get("sort")
+ .and_then(|s| s.as_array())
+ .filter(|a| !a.is_empty())
+ else {
+ warn!(
+ connector_id = self.id,
+ hit_id = hit.get("_id").and_then(|value| value.as_str()),
+ "Skipping OpenSearch hit without sort tuple; document will not be published"
+ );
+ continue;
+ };
+
+ last_sort = Some(sort);
+
+ let Some(source) = hit.get("_source") else {
+ warn!(
+ connector_id = self.id,
+ hit_id = hit.get("_id").and_then(|v| v.as_str()),
+ "Skipping OpenSearch hit without _source; document will not be published"
+ );
+ continue;
+ };
Review Comment:
can a document be empty in OpenSearch, or should the source be erroring out here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "kriti-sc (via GitHub)" <gi...@apache.org>.
kriti-sc commented on code in PR #3515:
URL: https://github.com/apache/iggy/pull/3515#discussion_r3521295741
##########
core/connectors/sources/opensearch_source/src/lib.rs:
##########
@@ -0,0 +1,1135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::retry::{CircuitBreaker, parse_duration};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use opensearch::{
+ OpenSearch, SearchParts,
+ auth::Credentials,
+ http::{Url, transport::TransportBuilder},
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use serde_json::{Value, json};
+use std::sync::Arc;
+use std::time::Duration;
+use tokio::{sync::Mutex, time::sleep};
+use tracing::{debug, error, info, warn};
+
+mod retry;
+mod state_manager;
+use crate::retry::{
+ DEFAULT_CB_COOL_DOWN, DEFAULT_CB_THRESHOLD, DEFAULT_MAX_OPEN_RETRIES, DEFAULT_MAX_RETRIES,
+ DEFAULT_OPEN_RETRY_MAX_DELAY, DEFAULT_RETRY_DELAY, DEFAULT_RETRY_MAX_DELAY, RetryBackoff,
+ is_transient_status, normalized_max_attempts, sleep_before_retry,
+};
+use crate::state_manager::{SOURCE_STATE_VERSION, SourceState, validate_state_storage_config};
+
+source_connector!(OpenSearchSource);
+
+const CONNECTOR_NAME: &str = "OpenSearch source";
+const DEFAULT_POLLING_INTERVAL: &str = "10s";
+const DEFAULT_BATCH_SIZE: usize = 100;
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct State {
+ #[serde(default)]
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ #[serde(default, alias = "total_documents_fetched")]
+ total_documents_published: usize,
+ #[serde(default)]
+ poll_count: usize,
+ /// OpenSearch `search_after` tuple from the last hit in the previous batch.
+ #[serde(default)]
+ search_after: Option<Vec<Value>>,
+ #[serde(default)]
+ error_count: usize,
+ #[serde(default)]
+ last_error: Option<String>,
+ #[serde(default)]
+ processing_stats: ProcessingStats,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct ProcessingStats {
+ #[serde(default)]
+ total_bytes_processed: u64,
+ /// Running cumulative average over the connector's lifetime, persisted and accumulated
+ /// across restarts. Reflects long-term throughput baseline, not session-only average.
+ #[serde(default)]
+ avg_batch_processing_time_ms: f64,
+ #[serde(default)]
+ last_successful_poll: Option<DateTime<Utc>>,
+ #[serde(default)]
+ empty_polls_count: usize,
+ #[serde(default)]
+ successful_polls_count: usize,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct StateConfig {
+ #[serde(default)]
+ pub enabled: bool,
+ pub storage_type: Option<String>,
+ pub storage_config: Option<Value>,
+ pub state_id: Option<String>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct OpenSearchSourceConfig {
+ pub url: String,
+ pub index: String,
+ pub username: Option<String>,
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_optional_secret")]
+ pub password: Option<SecretString>,
+ pub query: Option<Value>,
+ pub polling_interval: Option<String>,
+ pub batch_size: Option<usize>,
+ pub timestamp_field: Option<String>,
+ #[serde(default)]
+ pub verbose_logging: bool,
+ pub state: Option<StateConfig>,
+ /// Total poll-phase attempt count (not retry count); 1 = no retries, 3 = 2 retries.
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+ pub retry_max_delay: Option<String>,
+ /// Total open-phase attempt count (not retry count); 1 = no retries, 5 = 4 retries.
+ pub max_open_retries: Option<u32>,
+ pub open_retry_max_delay: Option<String>,
+ pub circuit_breaker_threshold: Option<u32>,
+ pub circuit_breaker_cool_down: Option<String>,
+}
+
+#[derive(Debug)]
+pub struct OpenSearchSource {
+ id: u32,
+ config: OpenSearchSourceConfig,
+ client: Option<OpenSearch>,
+ polling_interval: Duration,
+ /// Pre-built query body (query + size + sort). Set once in `open()` after config validation.
+ /// `None` before `open()` and after a failed `open()`.
+ search_body_base: Option<Value>,
+ verbose: bool,
+ max_retries: u32,
+ retry_delay: Duration,
+ retry_max_delay: Duration,
+ max_open_retries: u32,
+ open_retry_max_delay: Duration,
+ circuit_breaker: Arc<CircuitBreaker>,
+ state: Mutex<State>,
+ /// `Some(cause)` when runtime state restore was rejected; `None` means restore succeeded.
+ state_restore_error: Option<String>,
+ /// True when `new()` restored a valid runtime `ConnectorState`. File mirror must not override it.
+ runtime_state_restored: bool,
+}
+
+struct SearchOutcome {
+ messages: Vec<ProducedMessage>,
+ search_after: Option<Vec<Value>>,
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ batch_bytes: u64,
+}
+
+impl OpenSearchSource {
+ pub fn new(id: u32, config: OpenSearchSourceConfig, state: Option<ConnectorState>) -> Self {
+ let polling_interval =
+ parse_duration(config.polling_interval.as_deref(), DEFAULT_POLLING_INTERVAL);
+ let verbose = config.verbose_logging;
+ let (restored_state, state_restore_error, runtime_state_restored) =
+ restore_state(id, state);
+
+ let cb_threshold = config
+ .circuit_breaker_threshold
+ .unwrap_or(DEFAULT_CB_THRESHOLD);
+ let cb_cool_down = parse_duration(
+ config.circuit_breaker_cool_down.as_deref(),
+ DEFAULT_CB_COOL_DOWN,
+ );
+ let circuit_breaker = Arc::new(CircuitBreaker::new(cb_threshold, cb_cool_down));
+ let max_retries =
+ normalized_max_attempts(config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES));
+ let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+ let retry_max_delay =
+ parse_duration(config.retry_max_delay.as_deref(), DEFAULT_RETRY_MAX_DELAY);
+ let max_open_retries =
+ normalized_max_attempts(config.max_open_retries.unwrap_or(DEFAULT_MAX_OPEN_RETRIES));
+ let open_retry_max_delay = parse_duration(
+ config.open_retry_max_delay.as_deref(),
+ DEFAULT_OPEN_RETRY_MAX_DELAY,
+ );
+
+ OpenSearchSource {
+ id,
+ config,
+ max_retries,
+ retry_delay,
+ retry_max_delay,
+ max_open_retries,
+ open_retry_max_delay,
+ circuit_breaker,
+ client: None,
+ polling_interval,
+ search_body_base: None,
+ verbose,
+ state: Mutex::new(restored_state),
+ state_restore_error,
+ runtime_state_restored,
+ }
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn batch_size(&self) -> usize {
+ self.config.batch_size.unwrap_or(DEFAULT_BATCH_SIZE)
+ }
+
+ fn timestamp_field(&self) -> Result<&str, Error> {
+ self.config
+ .timestamp_field
+ .as_deref()
+ .filter(|s| !s.is_empty())
+ .ok_or_else(|| Error::InvalidConfigValue("timestamp_field is required".to_string()))
+ }
+
+ fn get_state_id(&self) -> String {
+ self.config
+ .state
+ .as_ref()
+ .and_then(|s| s.state_id.clone())
+ .unwrap_or_else(|| format!("opensearch_source_{}", self.id))
+ }
+
+ pub(crate) async fn internal_state_to_source_state(&self) -> Result<SourceState, Error> {
+ let state = self.state.lock().await;
+ let data = serde_json::to_value(&*state).map_err(|error| {
+ Error::Serialization(format!("Failed to serialize connector state: {error}"))
+ })?;
+
+ Ok(SourceState {
+ id: self.get_state_id(),
+ last_updated: Utc::now(),
+ version: SOURCE_STATE_VERSION,
+ data,
+ metadata: Some(json!({
+ "connector_type": "opensearch_source",
+ "connector_id": self.id,
+ "index": self.config.index,
+ })),
+ })
+ }
+
+ pub(crate) async fn source_state_to_internal_state(
+ &mut self,
+ source_state: SourceState,
+ ) -> Result<(), Error> {
+ if source_state.version != SOURCE_STATE_VERSION {
+ return Err(Error::Serialization(format!(
+ "unsupported file state version {}, expected {SOURCE_STATE_VERSION}",
+ source_state.version
+ )));
+ }
+
+ let restored: State = serde_json::from_value(source_state.data).map_err(|error| {
+ Error::Serialization(format!("Failed to deserialize connector state: {error}"))
+ })?;
+
+ let mut state = self.state.lock().await;
+ *state = restored;
+ Ok(())
+ }
+
+ async fn create_client(&self) -> Result<OpenSearch, Error> {
+ let url = Url::parse(&self.config.url).map_err(|error| {
+ Error::InvalidConfigValue(format!("Invalid OpenSearch URL: {error}"))
+ })?;
+
+ let conn_pool = opensearch::http::transport::SingleNodeConnectionPool::new(url);
+ let mut transport_builder = TransportBuilder::new(conn_pool);
+
+ if let (Some(username), Some(password)) = (&self.config.username, &self.config.password) {
+ let credentials =
+ Credentials::Basic(username.clone(), password.expose_secret().to_string());
+ transport_builder = transport_builder.auth(credentials);
+ }
+
+ let transport = transport_builder
+ .build()
+ .map_err(|error| Error::InitError(format!("Failed to build transport: {error}")))?;
+
+ Ok(OpenSearch::new(transport))
+ }
+
+ async fn check_index_exists_with_retry(&self, client: &OpenSearch) -> Result<(), Error> {
+ let max_attempts = self.max_open_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let response = match client
+ .indices()
+ .exists(opensearch::indices::IndicesExistsParts::Index(&[&self
+ .config
+ .index]))
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ // retry_delay is shared with the poll phase (no separate open-phase
+ // base delay); open_retry_max_delay caps the growth independently.
+ // Same pattern as the InfluxDB source connector.
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::InitError(format!(
+ "Failed to check index existence: {error}"
+ )));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(());
+ }
+
+ let status = response.status_code().as_u16();
+ if status == 404 {
+ return Err(Error::InitError(format!(
+ "Index '{}' does not exist or is not accessible",
+ self.config.index
+ )));
+ }
+
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::InitError(format!(
+ "Index '{}' check failed: HTTP {status}: {error_text}",
+ self.config.index
+ )));
+ }
+ }
+
+ async fn send_search_with_retry(
+ &self,
+ client: &OpenSearch,
+ mut search_body: Value,
+ ) -> Result<opensearch::http::response::Response, Error> {
+ let max_attempts = self.max_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let body = if attempt < max_attempts {
+ search_body.clone()
+ } else {
+ std::mem::take(&mut search_body)
+ };
+ let response = match client
+ .search(SearchParts::Index(&[&self.config.index]))
+ .body(body)
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::Storage(format!("Failed to execute search: {error}")));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(response);
+ }
+
+ let status = response.status_code().as_u16();
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::Storage(format!(
+ "Search request failed: {error_text}"
+ )));
+ }
+ }
+
+ async fn search_documents(&self, client: &OpenSearch) -> Result<SearchOutcome, Error> {
+ let state = self.state.lock().await;
+ let search_after = state.search_after.clone();
+ drop(state);
+
+ let timestamp_field = self.timestamp_field()?;
+
+ let mut search_body = self
+ .search_body_base
+ .as_ref()
+ .ok_or_else(|| {
+ Error::Connection("connector not initialized; call open() first".to_string())
+ })?
+ .clone();
+
+ if let Some(cursor) = search_after {
+ search_body["search_after"] = json!(cursor);
+ }
+
+ let response = self.send_search_with_retry(client, search_body).await?;
+
+ let mut response_body: Value = response
+ .json()
+ .await
+ .map_err(|e| Error::Storage(format!("Failed to parse search response: {e}")))?;
+
+ let hits: Vec<Value> = response_body
+ .get_mut("hits")
+ .and_then(|h| h.get_mut("hits"))
+ .and_then(|arr| arr.as_array_mut())
+ .map(std::mem::take)
+ .unwrap_or_default();
+
+ let mut messages = Vec::with_capacity(hits.len());
+ let mut batch_bytes = 0u64;
+ let mut last_sort = None;
+ let mut last_poll_timestamp = None;
+
+ for hit in &hits {
+ let Some(sort) = hit
+ .get("sort")
+ .and_then(|s| s.as_array())
+ .filter(|a| !a.is_empty())
+ else {
+ warn!(
+ connector_id = self.id,
+ hit_id = hit.get("_id").and_then(|value| value.as_str()),
+ "Skipping OpenSearch hit without sort tuple; document will not be published"
+ );
+ continue;
+ };
+
+ last_sort = Some(sort);
+
+ let Some(source) = hit.get("_source") else {
+ warn!(
+ connector_id = self.id,
+ hit_id = hit.get("_id").and_then(|v| v.as_str()),
+ "Skipping OpenSearch hit without _source; document will not be published"
+ );
+ continue;
+ };
+
+ if let Some(timestamp_value) = source.get(timestamp_field)
+ && let Some(timestamp_utc) = parse_document_timestamp(timestamp_value)
+ {
+ last_poll_timestamp = Some(timestamp_utc);
+ }
+
+ let payload = serde_json::to_vec(source)
+ .map_err(|e| Error::Serialization(format!("Failed to serialize document: {e}")))?;
+ batch_bytes += payload.len() as u64;
+
+ messages.push(ProducedMessage {
+ id: None,
+ headers: None,
+ checksum: None,
+ timestamp: None,
+ origin_timestamp: None,
+ payload,
+ });
+ }
+
+ if !hits.is_empty() && last_sort.is_none() {
+ return Err(Error::Storage(format!(
+ "OpenSearch returned {} hit(s) but none had a sort tuple; \
+ index may be missing the sort field or using an incompatible mapping",
+ hits.len()
+ )));
+ }
+
+ // Guard: cursor would advance past the entire batch but nothing would be published.
+ // This happens when _source is disabled on the index (all sort-bearing hits lack _source).
+ if !hits.is_empty() && messages.is_empty() && last_sort.is_some() {
+ return Err(Error::Storage(format!(
+ "OpenSearch returned {} hit(s) with valid sort tuples but all were missing \
+ _source; index may have _source disabled. Refusing to advance cursor \
+ without publishing any messages.",
+ hits.len()
+ )));
+ }
+
+ Ok(SearchOutcome {
+ messages,
+ search_after: last_sort.map(ToOwned::to_owned),
+ last_poll_timestamp,
+ batch_bytes,
+ })
+ }
+
+ async fn finalize_poll(
+ &self,
+ outcome: SearchOutcome,
+ processing_time_ms: f64,
+ ) -> (Vec<ProducedMessage>, Option<ConnectorState>) {
+ let mut state = self.state.lock().await;
+ state.total_documents_published += outcome.messages.len();
+ state.poll_count += 1;
+ if let Some(cursor) = outcome.search_after {
+ state.search_after = Some(cursor);
+ }
+ if let Some(timestamp) = outcome.last_poll_timestamp {
+ state.last_poll_timestamp = Some(timestamp);
+ }
+ state.processing_stats.total_bytes_processed += outcome.batch_bytes;
+
+ if outcome.messages.is_empty() {
+ state.processing_stats.empty_polls_count += 1;
+ } else {
+ state.processing_stats.successful_polls_count += 1;
+ state.processing_stats.last_successful_poll = Some(Utc::now());
+ }
+
+ let total_polls = state.processing_stats.successful_polls_count
+ + state.processing_stats.empty_polls_count;
+ // total_polls >= 1 because one of the two counters was just incremented above,
+ // but use saturating_sub to make the invariant machine-checked.
+ state.processing_stats.avg_batch_processing_time_ms =
+ (state.processing_stats.avg_batch_processing_time_ms
+ * total_polls.saturating_sub(1) as f64
+ + processing_time_ms)
+ / total_polls as f64;
+
+ let produced_count = outcome.messages.len();
+ let total_documents_published = state.total_documents_published;
+ let state_snapshot = state.clone();
+ let messages = outcome.messages;
+ drop(state);
+ let persisted_state = self.serialize_state(&state_snapshot);
+
+ if self.verbose {
+ info!(
+ "OpenSearch source connector ID: {} produced {produced_count} messages. \
+ Total published: {total_documents_published}",
+ self.id
+ );
+ } else {
+ debug!(
+ "OpenSearch source connector ID: {} produced {produced_count} messages. \
+ Total published: {total_documents_published}",
+ self.id
+ );
+ }
+
+ (messages, persisted_state)
+ }
+
+ #[cfg(test)]
+ fn client_initialized(&self) -> bool {
+ self.client.is_some()
+ }
+
+ #[cfg(test)]
+ async fn test_metrics(&self) -> (usize, usize, usize, usize) {
+ let state = self.state.lock().await;
+ (
+ state.total_documents_published,
+ state.poll_count,
+ state.error_count,
+ state.processing_stats.empty_polls_count,
+ )
+ }
+
+ #[cfg(test)]
+ async fn test_search_after(&self) -> Option<Vec<Value>> {
+ self.state.lock().await.search_after.clone()
+ }
+
+ #[cfg(test)]
+ async fn test_last_poll_timestamp(&self) -> Option<DateTime<Utc>> {
+ self.state.lock().await.last_poll_timestamp
+ }
+
+ async fn handle_poll_error(&self, error: Error) -> Result<ProducedMessages, Error> {
+ self.circuit_breaker.record_failure().await;
+ let mut state = self.state.lock().await;
+ // error_count and last_error accumulate in State and are captured by the next
+ // finalize_poll call (success path). Both runtime ConnectorState and file state
+ // preserve these values across restarts.
+ state.error_count += 1;
+ state.last_error = Some(error.to_string());
+ drop(state);
+ error!(
+ "{CONNECTOR_NAME} connector ID: {} poll failed: {error}",
+ self.id
+ );
+ Err(error)
+ }
+}
+
+fn restore_state(id: u32, state: Option<ConnectorState>) -> (State, Option<String>, bool) {
+ let Some(connector_state) = state else {
+ return (State::default(), None, false);
+ };
+
+ match connector_state.deserialize::<State>(CONNECTOR_NAME, id) {
+ Some(restored) => {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Documents published: {}, poll count: {}",
+ restored.total_documents_published, restored.poll_count
+ );
+ (restored, None, true)
+ }
+ None => {
+ let cause = "persisted state exists but could not be deserialized. \
+ Refusing to start to prevent silent cursor reset."
+ .to_string();
+ error!("{CONNECTOR_NAME} ID {id}: {cause}");
+ (State::default(), Some(cause), false)
+ }
+ }
+}
+
+fn validate_open_config(config: &OpenSearchSourceConfig) -> Result<(), Error> {
+ if config.timestamp_field.as_deref().is_none_or(str::is_empty) {
+ return Err(Error::InvalidConfigValue(
+ "timestamp_field is required for incremental OpenSearch polling".to_string(),
+ ));
+ }
+
+ if matches!(config.batch_size, Some(0)) {
+ return Err(Error::InvalidConfigValue(
+ "batch_size must be at least 1".to_string(),
+ ));
+ }
+
+ if let Some(state) = &config.state
+ && state.enabled
+ {
+ validate_state_storage_config(state)?;
+ }
+
+ Ok(())
+}
+
+fn parse_document_timestamp(value: &Value) -> Option<DateTime<Utc>> {
+ match value {
+ Value::String(text) => DateTime::parse_from_rfc3339(text)
+ .ok()
+ .map(|timestamp| timestamp.with_timezone(&Utc)),
+ Value::Number(number) => {
+ let raw = number.as_i64()?;
+ // Values above 1e12 are already milliseconds (Unix epoch seconds won't reach
+ // 1e12 until year 33658). Values at or below are treated as seconds and
+ // multiplied by 1000.
+ let millis = if !(-1_000_000_000_000..=1_000_000_000_000).contains(&raw) {
+ raw
+ } else {
+ raw.saturating_mul(1_000)
+ };
+ DateTime::from_timestamp_millis(millis).map(|timestamp| timestamp.with_timezone(&Utc))
+ }
+ _ => None,
+ }
+}
+
+#[async_trait]
+impl Source for OpenSearchSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ if let Some(ref cause) = self.state_restore_error {
+ return Err(Error::InitError(format!("state restore failed: {cause}")));
+ }
+
+ validate_open_config(&self.config)?;
+
+ info!(
+ "Opening OpenSearch source connector with ID: {} for URL: {}, index: {}",
+ self.id, self.config.url, self.config.index
+ );
+
+ let client = self.create_client().await?;
+
+ self.check_index_exists_with_retry(&client).await?;
+
+ let timestamp_field = self.timestamp_field()?;
+ let batch_size = self.batch_size();
+ self.search_body_base = Some(json!({
+ "query": self.config.query.clone().unwrap_or_else(|| json!({ "match_all": {} })),
+ "size": batch_size,
+ "sort": [
+ { timestamp_field: { "order": "asc" } },
+ { "_id": { "order": "asc" } }
+ ]
+ }));
Review Comment:
is _id global unique? ie can two docs have the same id? if not, this is a possible data loss issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #3515:
URL: https://github.com/apache/iggy/pull/3515#discussion_r3530518819
##########
core/connectors/sources/opensearch_source/src/lib.rs:
##########
@@ -0,0 +1,1135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::retry::{CircuitBreaker, parse_duration};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use opensearch::{
+ OpenSearch, SearchParts,
+ auth::Credentials,
+ http::{Url, transport::TransportBuilder},
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use serde_json::{Value, json};
+use std::sync::Arc;
+use std::time::Duration;
+use tokio::{sync::Mutex, time::sleep};
+use tracing::{debug, error, info, warn};
+
+mod retry;
+mod state_manager;
+use crate::retry::{
+ DEFAULT_CB_COOL_DOWN, DEFAULT_CB_THRESHOLD, DEFAULT_MAX_OPEN_RETRIES, DEFAULT_MAX_RETRIES,
+ DEFAULT_OPEN_RETRY_MAX_DELAY, DEFAULT_RETRY_DELAY, DEFAULT_RETRY_MAX_DELAY, RetryBackoff,
+ is_transient_status, normalized_max_attempts, sleep_before_retry,
+};
+use crate::state_manager::{SOURCE_STATE_VERSION, SourceState, validate_state_storage_config};
+
+source_connector!(OpenSearchSource);
+
+const CONNECTOR_NAME: &str = "OpenSearch source";
+const DEFAULT_POLLING_INTERVAL: &str = "10s";
+const DEFAULT_BATCH_SIZE: usize = 100;
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct State {
+ #[serde(default)]
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ #[serde(default, alias = "total_documents_fetched")]
+ total_documents_published: usize,
+ #[serde(default)]
+ poll_count: usize,
+ /// OpenSearch `search_after` tuple from the last hit in the previous batch.
+ #[serde(default)]
+ search_after: Option<Vec<Value>>,
+ #[serde(default)]
+ error_count: usize,
+ #[serde(default)]
+ last_error: Option<String>,
+ #[serde(default)]
+ processing_stats: ProcessingStats,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct ProcessingStats {
+ #[serde(default)]
+ total_bytes_processed: u64,
+ /// Running cumulative average over the connector's lifetime, persisted and accumulated
+ /// across restarts. Reflects long-term throughput baseline, not session-only average.
+ #[serde(default)]
+ avg_batch_processing_time_ms: f64,
+ #[serde(default)]
+ last_successful_poll: Option<DateTime<Utc>>,
+ #[serde(default)]
+ empty_polls_count: usize,
+ #[serde(default)]
+ successful_polls_count: usize,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct StateConfig {
+ #[serde(default)]
+ pub enabled: bool,
+ pub storage_type: Option<String>,
+ pub storage_config: Option<Value>,
+ pub state_id: Option<String>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct OpenSearchSourceConfig {
+ pub url: String,
+ pub index: String,
+ pub username: Option<String>,
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_optional_secret")]
+ pub password: Option<SecretString>,
+ pub query: Option<Value>,
+ pub polling_interval: Option<String>,
+ pub batch_size: Option<usize>,
+ pub timestamp_field: Option<String>,
+ #[serde(default)]
+ pub verbose_logging: bool,
+ pub state: Option<StateConfig>,
+ /// Total poll-phase attempt count (not retry count); 1 = no retries, 3 = 2 retries.
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+ pub retry_max_delay: Option<String>,
+ /// Total open-phase attempt count (not retry count); 1 = no retries, 5 = 4 retries.
+ pub max_open_retries: Option<u32>,
+ pub open_retry_max_delay: Option<String>,
+ pub circuit_breaker_threshold: Option<u32>,
+ pub circuit_breaker_cool_down: Option<String>,
+}
+
+#[derive(Debug)]
+pub struct OpenSearchSource {
+ id: u32,
+ config: OpenSearchSourceConfig,
+ client: Option<OpenSearch>,
+ polling_interval: Duration,
+ /// Pre-built query body (query + size + sort). Set once in `open()` after config validation.
+ /// `None` before `open()` and after a failed `open()`.
+ search_body_base: Option<Value>,
+ verbose: bool,
+ max_retries: u32,
+ retry_delay: Duration,
+ retry_max_delay: Duration,
+ max_open_retries: u32,
+ open_retry_max_delay: Duration,
+ circuit_breaker: Arc<CircuitBreaker>,
+ state: Mutex<State>,
+ /// `Some(cause)` when runtime state restore was rejected; `None` means restore succeeded.
+ state_restore_error: Option<String>,
+ /// True when `new()` restored a valid runtime `ConnectorState`. File mirror must not override it.
+ runtime_state_restored: bool,
+}
+
+struct SearchOutcome {
+ messages: Vec<ProducedMessage>,
+ search_after: Option<Vec<Value>>,
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ batch_bytes: u64,
+}
+
+impl OpenSearchSource {
+ pub fn new(id: u32, config: OpenSearchSourceConfig, state: Option<ConnectorState>) -> Self {
+ let polling_interval =
+ parse_duration(config.polling_interval.as_deref(), DEFAULT_POLLING_INTERVAL);
+ let verbose = config.verbose_logging;
+ let (restored_state, state_restore_error, runtime_state_restored) =
+ restore_state(id, state);
+
+ let cb_threshold = config
+ .circuit_breaker_threshold
+ .unwrap_or(DEFAULT_CB_THRESHOLD);
+ let cb_cool_down = parse_duration(
+ config.circuit_breaker_cool_down.as_deref(),
+ DEFAULT_CB_COOL_DOWN,
+ );
+ let circuit_breaker = Arc::new(CircuitBreaker::new(cb_threshold, cb_cool_down));
+ let max_retries =
+ normalized_max_attempts(config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES));
+ let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+ let retry_max_delay =
+ parse_duration(config.retry_max_delay.as_deref(), DEFAULT_RETRY_MAX_DELAY);
+ let max_open_retries =
+ normalized_max_attempts(config.max_open_retries.unwrap_or(DEFAULT_MAX_OPEN_RETRIES));
+ let open_retry_max_delay = parse_duration(
+ config.open_retry_max_delay.as_deref(),
+ DEFAULT_OPEN_RETRY_MAX_DELAY,
+ );
+
+ OpenSearchSource {
+ id,
+ config,
+ max_retries,
+ retry_delay,
+ retry_max_delay,
+ max_open_retries,
+ open_retry_max_delay,
+ circuit_breaker,
+ client: None,
+ polling_interval,
+ search_body_base: None,
+ verbose,
+ state: Mutex::new(restored_state),
+ state_restore_error,
+ runtime_state_restored,
+ }
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn batch_size(&self) -> usize {
+ self.config.batch_size.unwrap_or(DEFAULT_BATCH_SIZE)
+ }
+
+ fn timestamp_field(&self) -> Result<&str, Error> {
+ self.config
+ .timestamp_field
+ .as_deref()
+ .filter(|s| !s.is_empty())
+ .ok_or_else(|| Error::InvalidConfigValue("timestamp_field is required".to_string()))
+ }
+
+ fn get_state_id(&self) -> String {
+ self.config
+ .state
+ .as_ref()
+ .and_then(|s| s.state_id.clone())
+ .unwrap_or_else(|| format!("opensearch_source_{}", self.id))
+ }
+
+ pub(crate) async fn internal_state_to_source_state(&self) -> Result<SourceState, Error> {
+ let state = self.state.lock().await;
+ let data = serde_json::to_value(&*state).map_err(|error| {
+ Error::Serialization(format!("Failed to serialize connector state: {error}"))
+ })?;
+
+ Ok(SourceState {
+ id: self.get_state_id(),
+ last_updated: Utc::now(),
+ version: SOURCE_STATE_VERSION,
+ data,
+ metadata: Some(json!({
+ "connector_type": "opensearch_source",
+ "connector_id": self.id,
+ "index": self.config.index,
+ })),
+ })
+ }
+
+ pub(crate) async fn source_state_to_internal_state(
+ &mut self,
+ source_state: SourceState,
+ ) -> Result<(), Error> {
+ if source_state.version != SOURCE_STATE_VERSION {
+ return Err(Error::Serialization(format!(
+ "unsupported file state version {}, expected {SOURCE_STATE_VERSION}",
+ source_state.version
+ )));
+ }
+
+ let restored: State = serde_json::from_value(source_state.data).map_err(|error| {
+ Error::Serialization(format!("Failed to deserialize connector state: {error}"))
+ })?;
+
+ let mut state = self.state.lock().await;
+ *state = restored;
+ Ok(())
+ }
+
+ async fn create_client(&self) -> Result<OpenSearch, Error> {
+ let url = Url::parse(&self.config.url).map_err(|error| {
+ Error::InvalidConfigValue(format!("Invalid OpenSearch URL: {error}"))
+ })?;
+
+ let conn_pool = opensearch::http::transport::SingleNodeConnectionPool::new(url);
+ let mut transport_builder = TransportBuilder::new(conn_pool);
+
+ if let (Some(username), Some(password)) = (&self.config.username, &self.config.password) {
+ let credentials =
+ Credentials::Basic(username.clone(), password.expose_secret().to_string());
+ transport_builder = transport_builder.auth(credentials);
+ }
+
+ let transport = transport_builder
+ .build()
+ .map_err(|error| Error::InitError(format!("Failed to build transport: {error}")))?;
+
+ Ok(OpenSearch::new(transport))
+ }
+
+ async fn check_index_exists_with_retry(&self, client: &OpenSearch) -> Result<(), Error> {
+ let max_attempts = self.max_open_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let response = match client
+ .indices()
+ .exists(opensearch::indices::IndicesExistsParts::Index(&[&self
+ .config
+ .index]))
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ // retry_delay is shared with the poll phase (no separate open-phase
+ // base delay); open_retry_max_delay caps the growth independently.
+ // Same pattern as the InfluxDB source connector.
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::InitError(format!(
+ "Failed to check index existence: {error}"
+ )));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(());
+ }
+
+ let status = response.status_code().as_u16();
+ if status == 404 {
+ return Err(Error::InitError(format!(
+ "Index '{}' does not exist or is not accessible",
+ self.config.index
+ )));
+ }
+
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::InitError(format!(
+ "Index '{}' check failed: HTTP {status}: {error_text}",
+ self.config.index
+ )));
+ }
+ }
+
+ async fn send_search_with_retry(
+ &self,
+ client: &OpenSearch,
+ mut search_body: Value,
+ ) -> Result<opensearch::http::response::Response, Error> {
+ let max_attempts = self.max_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let body = if attempt < max_attempts {
+ search_body.clone()
+ } else {
+ std::mem::take(&mut search_body)
+ };
+ let response = match client
+ .search(SearchParts::Index(&[&self.config.index]))
+ .body(body)
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::Storage(format!("Failed to execute search: {error}")));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(response);
+ }
+
+ let status = response.status_code().as_u16();
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::Storage(format!(
+ "Search request failed: {error_text}"
+ )));
+ }
+ }
+
+ async fn search_documents(&self, client: &OpenSearch) -> Result<SearchOutcome, Error> {
+ let state = self.state.lock().await;
+ let search_after = state.search_after.clone();
+ drop(state);
+
+ let timestamp_field = self.timestamp_field()?;
+
+ let mut search_body = self
+ .search_body_base
+ .as_ref()
+ .ok_or_else(|| {
+ Error::Connection("connector not initialized; call open() first".to_string())
+ })?
+ .clone();
+
+ if let Some(cursor) = search_after {
+ search_body["search_after"] = json!(cursor);
+ }
+
+ let response = self.send_search_with_retry(client, search_body).await?;
+
+ let mut response_body: Value = response
+ .json()
+ .await
+ .map_err(|e| Error::Storage(format!("Failed to parse search response: {e}")))?;
+
+ let hits: Vec<Value> = response_body
+ .get_mut("hits")
+ .and_then(|h| h.get_mut("hits"))
+ .and_then(|arr| arr.as_array_mut())
+ .map(std::mem::take)
+ .unwrap_or_default();
+
+ let mut messages = Vec::with_capacity(hits.len());
+ let mut batch_bytes = 0u64;
+ let mut last_sort = None;
+ let mut last_poll_timestamp = None;
+
+ for hit in &hits {
+ let Some(sort) = hit
+ .get("sort")
+ .and_then(|s| s.as_array())
+ .filter(|a| !a.is_empty())
+ else {
+ warn!(
+ connector_id = self.id,
+ hit_id = hit.get("_id").and_then(|value| value.as_str()),
+ "Skipping OpenSearch hit without sort tuple; document will not be published"
+ );
+ continue;
+ };
+
+ last_sort = Some(sort);
Review Comment:
how would the test case help - I am habituated with writing the use cases, test cases then start writing the code - TDD. Thinking of different data set scenarios and mix of data is one of the criteria in the Igyy connectors development. By writing test cases with mix of data so that cursor is handles the bad data and continues with the expected behavior. This is the reason, my source contributions have a higher percentage of code coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3515:
URL: https://github.com/apache/iggy/pull/3515#issuecomment-5486868574
This pull request was automatically closed because it has been inactive for 14 days. Feel free to reopen it if you'd like to continue working on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3515: feat(connectors): opensearch source connector
URL: https://github.com/apache/iggy/pull/3515
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #3515:
URL: https://github.com/apache/iggy/pull/3515#discussion_r3522494152
##########
core/connectors/sources/opensearch_source/src/state_manager.rs:
##########
@@ -0,0 +1,314 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::{OpenSearchSource, StateConfig};
+use async_trait::async_trait;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::Error;
+use serde::{Deserialize, Serialize};
+use std::io::ErrorKind;
+use std::sync::Arc;
+use tracing::info;
+
+pub(crate) const SOURCE_STATE_VERSION: u32 = 1;
+
+impl OpenSearchSource {
+ pub(super) async fn save_state(&self) -> Result<(), Error> {
+ let Some(state_config) = self.config.state.as_ref().filter(|s| s.enabled) else {
+ return Ok(());
+ };
+ let storage = create_state_storage(state_config)?;
+
+ let source_state = self.internal_state_to_source_state().await?;
+ storage.save_source_state(&source_state).await?;
+
+ info!(
+ "Saved state for OpenSearch source connector with ID: {}",
+ self.id
+ );
+ Ok(())
+ }
+
+ pub(super) async fn load_state(&mut self) -> Result<(), Error> {
+ let Some(state_config) = self.config.state.as_ref().filter(|s| s.enabled) else {
+ return Ok(());
+ };
+ let storage = create_state_storage(state_config)?;
+
+ let state_id = self.get_state_id();
+ if let Some(source_state) = storage.load_source_state(&state_id).await? {
+ self.source_state_to_internal_state(source_state).await?;
+
+ let (last_poll_timestamp, total_documents_published, poll_count) = {
+ let state = self.state.lock().await;
+ (
+ state.last_poll_timestamp,
+ state.total_documents_published,
+ state.poll_count,
+ )
+ };
+ info!(
+ "Loaded state for OpenSearch source connector with ID: {} - last poll: {:?}, total docs: {}, polls: {}",
+ self.id, last_poll_timestamp, total_documents_published, poll_count
+ );
+ } else {
+ info!(
+ "No existing state found for OpenSearch source connector with ID: {}, starting fresh",
+ self.id
+ );
+ }
+
+ Ok(())
+ }
+}
+
+pub(crate) fn validate_state_storage_config(config: &StateConfig) -> Result<(), Error> {
+ match config.storage_type.as_deref() {
+ Some("file") | None => Ok(()),
+ Some(storage_type) => Err(Error::InvalidConfigValue(format!(
+ "state storage_type {storage_type:?} is not supported; only \"file\" is implemented"
+ ))),
+ }
+}
+
+pub(crate) fn create_state_storage(config: &StateConfig) -> Result<Arc<dyn StateStorage>, Error> {
+ let base_path = config
+ .storage_config
+ .as_ref()
+ .and_then(|c| c.get("base_path"))
+ .and_then(|p| p.as_str())
+ .unwrap_or("./connector_states");
+
+ Ok(Arc::new(FileStateStorage::new(base_path)))
+}
+
+/// Optional file-backed mirror of connector state. Runtime `ConnectorState` msgpack is authoritative.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub(crate) struct SourceState {
+ pub id: String,
+ pub last_updated: DateTime<Utc>,
+ #[serde(default)]
+ pub version: u32,
+ pub data: serde_json::Value,
+ pub metadata: Option<serde_json::Value>,
+}
+
+#[async_trait]
+pub(crate) trait StateStorage: Send + Sync {
+ async fn save_source_state(&self, state: &SourceState) -> Result<(), Error>;
+
+ async fn load_source_state(&self, id: &str) -> Result<Option<SourceState>, Error>;
+}
+
+pub(crate) struct FileStateStorage {
+ base_path: std::path::PathBuf,
+}
+
+impl FileStateStorage {
+ pub(crate) fn new<P: AsRef<std::path::Path>>(base_path: P) -> Self {
+ Self {
+ base_path: base_path.as_ref().to_path_buf(),
+ }
+ }
+
+ fn get_state_path(&self, id: &str) -> std::path::PathBuf {
+ self.base_path.join(format!("{id}.json"))
+ }
+}
+
+#[async_trait]
+impl StateStorage for FileStateStorage {
+ async fn save_source_state(&self, state: &SourceState) -> Result<(), Error> {
+ use tokio::fs::{self, OpenOptions};
+ use tokio::io::AsyncWriteExt;
+
+ fs::create_dir_all(&self.base_path)
+ .await
+ .map_err(|e| Error::Storage(format!("Failed to create state directory: {e}")))?;
+
+ let path = self.get_state_path(&state.id);
+ // PID in the suffix avoids races when two processes write the same state_id.
+ let tmp_path = path.with_extension(format!("json.{}.tmp", std::process::id()));
+ let json = serde_json::to_string(state)
+ .map_err(|e| Error::Serialization(format!("Failed to serialize source state: {e}")))?;
+
+ let mut tmp_file = OpenOptions::new()
+ .write(true)
+ .create(true)
+ .truncate(true)
+ .open(&tmp_path)
+ .await
+ .map_err(|e| Error::Storage(format!("Failed to open state temp file: {e}")))?;
+ if let Err(e) = tmp_file.write_all(json.as_bytes()).await {
+ let _ = fs::remove_file(&tmp_path).await;
+ return Err(Error::Storage(format!(
+ "Failed to write state temp file: {e}"
+ )));
+ }
Review Comment:
Good catch. Let me borrow from core/connectors/runtime/src/state.rs (168-174) . It provides 0o600 and probably can give the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on PR #3515:
URL: https://github.com/apache/iggy/pull/3515#issuecomment-4882173202
/ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "pushp-ranja (via GitHub)" <gi...@apache.org>.
pushp-ranja commented on code in PR #3515:
URL: https://github.com/apache/iggy/pull/3515#discussion_r3524933218
##########
core/connectors/sources/opensearch_source/src/lib.rs:
##########
@@ -0,0 +1,1135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::retry::{CircuitBreaker, parse_duration};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use opensearch::{
+ OpenSearch, SearchParts,
+ auth::Credentials,
+ http::{Url, transport::TransportBuilder},
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use serde_json::{Value, json};
+use std::sync::Arc;
+use std::time::Duration;
+use tokio::{sync::Mutex, time::sleep};
+use tracing::{debug, error, info, warn};
+
+mod retry;
+mod state_manager;
+use crate::retry::{
+ DEFAULT_CB_COOL_DOWN, DEFAULT_CB_THRESHOLD, DEFAULT_MAX_OPEN_RETRIES, DEFAULT_MAX_RETRIES,
+ DEFAULT_OPEN_RETRY_MAX_DELAY, DEFAULT_RETRY_DELAY, DEFAULT_RETRY_MAX_DELAY, RetryBackoff,
+ is_transient_status, normalized_max_attempts, sleep_before_retry,
+};
+use crate::state_manager::{SOURCE_STATE_VERSION, SourceState, validate_state_storage_config};
+
+source_connector!(OpenSearchSource);
+
+const CONNECTOR_NAME: &str = "OpenSearch source";
+const DEFAULT_POLLING_INTERVAL: &str = "10s";
+const DEFAULT_BATCH_SIZE: usize = 100;
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct State {
+ #[serde(default)]
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ #[serde(default, alias = "total_documents_fetched")]
+ total_documents_published: usize,
+ #[serde(default)]
+ poll_count: usize,
+ /// OpenSearch `search_after` tuple from the last hit in the previous batch.
+ #[serde(default)]
+ search_after: Option<Vec<Value>>,
+ #[serde(default)]
+ error_count: usize,
+ #[serde(default)]
+ last_error: Option<String>,
+ #[serde(default)]
+ processing_stats: ProcessingStats,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct ProcessingStats {
+ #[serde(default)]
+ total_bytes_processed: u64,
+ /// Running cumulative average over the connector's lifetime, persisted and accumulated
+ /// across restarts. Reflects long-term throughput baseline, not session-only average.
+ #[serde(default)]
+ avg_batch_processing_time_ms: f64,
+ #[serde(default)]
+ last_successful_poll: Option<DateTime<Utc>>,
+ #[serde(default)]
+ empty_polls_count: usize,
+ #[serde(default)]
+ successful_polls_count: usize,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct StateConfig {
+ #[serde(default)]
+ pub enabled: bool,
+ pub storage_type: Option<String>,
+ pub storage_config: Option<Value>,
+ pub state_id: Option<String>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct OpenSearchSourceConfig {
+ pub url: String,
+ pub index: String,
+ pub username: Option<String>,
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_optional_secret")]
+ pub password: Option<SecretString>,
+ pub query: Option<Value>,
+ pub polling_interval: Option<String>,
+ pub batch_size: Option<usize>,
+ pub timestamp_field: Option<String>,
+ #[serde(default)]
+ pub verbose_logging: bool,
+ pub state: Option<StateConfig>,
+ /// Total poll-phase attempt count (not retry count); 1 = no retries, 3 = 2 retries.
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+ pub retry_max_delay: Option<String>,
+ /// Total open-phase attempt count (not retry count); 1 = no retries, 5 = 4 retries.
+ pub max_open_retries: Option<u32>,
+ pub open_retry_max_delay: Option<String>,
+ pub circuit_breaker_threshold: Option<u32>,
+ pub circuit_breaker_cool_down: Option<String>,
+}
+
+#[derive(Debug)]
+pub struct OpenSearchSource {
+ id: u32,
+ config: OpenSearchSourceConfig,
+ client: Option<OpenSearch>,
+ polling_interval: Duration,
+ /// Pre-built query body (query + size + sort). Set once in `open()` after config validation.
+ /// `None` before `open()` and after a failed `open()`.
+ search_body_base: Option<Value>,
+ verbose: bool,
+ max_retries: u32,
+ retry_delay: Duration,
+ retry_max_delay: Duration,
+ max_open_retries: u32,
+ open_retry_max_delay: Duration,
+ circuit_breaker: Arc<CircuitBreaker>,
+ state: Mutex<State>,
+ /// `Some(cause)` when runtime state restore was rejected; `None` means restore succeeded.
+ state_restore_error: Option<String>,
+ /// True when `new()` restored a valid runtime `ConnectorState`. File mirror must not override it.
+ runtime_state_restored: bool,
+}
+
+struct SearchOutcome {
+ messages: Vec<ProducedMessage>,
+ search_after: Option<Vec<Value>>,
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ batch_bytes: u64,
+}
+
+impl OpenSearchSource {
+ pub fn new(id: u32, config: OpenSearchSourceConfig, state: Option<ConnectorState>) -> Self {
+ let polling_interval =
+ parse_duration(config.polling_interval.as_deref(), DEFAULT_POLLING_INTERVAL);
+ let verbose = config.verbose_logging;
+ let (restored_state, state_restore_error, runtime_state_restored) =
+ restore_state(id, state);
+
+ let cb_threshold = config
+ .circuit_breaker_threshold
+ .unwrap_or(DEFAULT_CB_THRESHOLD);
+ let cb_cool_down = parse_duration(
+ config.circuit_breaker_cool_down.as_deref(),
+ DEFAULT_CB_COOL_DOWN,
+ );
+ let circuit_breaker = Arc::new(CircuitBreaker::new(cb_threshold, cb_cool_down));
+ let max_retries =
+ normalized_max_attempts(config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES));
+ let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+ let retry_max_delay =
+ parse_duration(config.retry_max_delay.as_deref(), DEFAULT_RETRY_MAX_DELAY);
+ let max_open_retries =
+ normalized_max_attempts(config.max_open_retries.unwrap_or(DEFAULT_MAX_OPEN_RETRIES));
+ let open_retry_max_delay = parse_duration(
+ config.open_retry_max_delay.as_deref(),
+ DEFAULT_OPEN_RETRY_MAX_DELAY,
+ );
+
+ OpenSearchSource {
+ id,
+ config,
+ max_retries,
+ retry_delay,
+ retry_max_delay,
+ max_open_retries,
+ open_retry_max_delay,
+ circuit_breaker,
+ client: None,
+ polling_interval,
+ search_body_base: None,
+ verbose,
+ state: Mutex::new(restored_state),
+ state_restore_error,
+ runtime_state_restored,
+ }
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn batch_size(&self) -> usize {
+ self.config.batch_size.unwrap_or(DEFAULT_BATCH_SIZE)
+ }
+
+ fn timestamp_field(&self) -> Result<&str, Error> {
+ self.config
+ .timestamp_field
+ .as_deref()
+ .filter(|s| !s.is_empty())
+ .ok_or_else(|| Error::InvalidConfigValue("timestamp_field is required".to_string()))
+ }
+
+ fn get_state_id(&self) -> String {
+ self.config
+ .state
+ .as_ref()
+ .and_then(|s| s.state_id.clone())
+ .unwrap_or_else(|| format!("opensearch_source_{}", self.id))
+ }
+
+ pub(crate) async fn internal_state_to_source_state(&self) -> Result<SourceState, Error> {
+ let state = self.state.lock().await;
+ let data = serde_json::to_value(&*state).map_err(|error| {
+ Error::Serialization(format!("Failed to serialize connector state: {error}"))
+ })?;
+
+ Ok(SourceState {
+ id: self.get_state_id(),
+ last_updated: Utc::now(),
+ version: SOURCE_STATE_VERSION,
+ data,
+ metadata: Some(json!({
+ "connector_type": "opensearch_source",
+ "connector_id": self.id,
+ "index": self.config.index,
+ })),
+ })
+ }
+
+ pub(crate) async fn source_state_to_internal_state(
+ &mut self,
+ source_state: SourceState,
+ ) -> Result<(), Error> {
+ if source_state.version != SOURCE_STATE_VERSION {
+ return Err(Error::Serialization(format!(
+ "unsupported file state version {}, expected {SOURCE_STATE_VERSION}",
+ source_state.version
+ )));
+ }
+
+ let restored: State = serde_json::from_value(source_state.data).map_err(|error| {
+ Error::Serialization(format!("Failed to deserialize connector state: {error}"))
+ })?;
+
+ let mut state = self.state.lock().await;
+ *state = restored;
+ Ok(())
+ }
+
+ async fn create_client(&self) -> Result<OpenSearch, Error> {
+ let url = Url::parse(&self.config.url).map_err(|error| {
+ Error::InvalidConfigValue(format!("Invalid OpenSearch URL: {error}"))
+ })?;
+
+ let conn_pool = opensearch::http::transport::SingleNodeConnectionPool::new(url);
+ let mut transport_builder = TransportBuilder::new(conn_pool);
+
+ if let (Some(username), Some(password)) = (&self.config.username, &self.config.password) {
+ let credentials =
+ Credentials::Basic(username.clone(), password.expose_secret().to_string());
+ transport_builder = transport_builder.auth(credentials);
+ }
+
+ let transport = transport_builder
+ .build()
+ .map_err(|error| Error::InitError(format!("Failed to build transport: {error}")))?;
+
+ Ok(OpenSearch::new(transport))
+ }
+
+ async fn check_index_exists_with_retry(&self, client: &OpenSearch) -> Result<(), Error> {
+ let max_attempts = self.max_open_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let response = match client
+ .indices()
+ .exists(opensearch::indices::IndicesExistsParts::Index(&[&self
+ .config
+ .index]))
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ // retry_delay is shared with the poll phase (no separate open-phase
+ // base delay); open_retry_max_delay caps the growth independently.
+ // Same pattern as the InfluxDB source connector.
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::InitError(format!(
+ "Failed to check index existence: {error}"
+ )));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(());
+ }
+
+ let status = response.status_code().as_u16();
+ if status == 404 {
+ return Err(Error::InitError(format!(
+ "Index '{}' does not exist or is not accessible",
+ self.config.index
+ )));
+ }
+
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::InitError(format!(
+ "Index '{}' check failed: HTTP {status}: {error_text}",
+ self.config.index
+ )));
+ }
+ }
+
+ async fn send_search_with_retry(
+ &self,
+ client: &OpenSearch,
+ mut search_body: Value,
+ ) -> Result<opensearch::http::response::Response, Error> {
+ let max_attempts = self.max_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let body = if attempt < max_attempts {
+ search_body.clone()
+ } else {
+ std::mem::take(&mut search_body)
+ };
+ let response = match client
+ .search(SearchParts::Index(&[&self.config.index]))
+ .body(body)
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::Storage(format!("Failed to execute search: {error}")));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(response);
+ }
+
+ let status = response.status_code().as_u16();
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::Storage(format!(
+ "Search request failed: {error_text}"
+ )));
+ }
+ }
+
+ async fn search_documents(&self, client: &OpenSearch) -> Result<SearchOutcome, Error> {
+ let state = self.state.lock().await;
+ let search_after = state.search_after.clone();
+ drop(state);
+
+ let timestamp_field = self.timestamp_field()?;
+
+ let mut search_body = self
+ .search_body_base
+ .as_ref()
+ .ok_or_else(|| {
+ Error::Connection("connector not initialized; call open() first".to_string())
+ })?
+ .clone();
+
+ if let Some(cursor) = search_after {
+ search_body["search_after"] = json!(cursor);
+ }
+
+ let response = self.send_search_with_retry(client, search_body).await?;
+
+ let mut response_body: Value = response
+ .json()
+ .await
+ .map_err(|e| Error::Storage(format!("Failed to parse search response: {e}")))?;
+
+ let hits: Vec<Value> = response_body
+ .get_mut("hits")
+ .and_then(|h| h.get_mut("hits"))
+ .and_then(|arr| arr.as_array_mut())
+ .map(std::mem::take)
+ .unwrap_or_default();
+
+ let mut messages = Vec::with_capacity(hits.len());
+ let mut batch_bytes = 0u64;
+ let mut last_sort = None;
+ let mut last_poll_timestamp = None;
+
+ for hit in &hits {
+ let Some(sort) = hit
+ .get("sort")
+ .and_then(|s| s.as_array())
+ .filter(|a| !a.is_empty())
+ else {
+ warn!(
+ connector_id = self.id,
+ hit_id = hit.get("_id").and_then(|value| value.as_str()),
+ "Skipping OpenSearch hit without sort tuple; document will not be published"
+ );
+ continue;
+ };
+
+ last_sort = Some(sort);
+
+ let Some(source) = hit.get("_source") else {
+ warn!(
+ connector_id = self.id,
+ hit_id = hit.get("_id").and_then(|v| v.as_str()),
+ "Skipping OpenSearch hit without _source; document will not be published"
+ );
+ continue;
+ };
Review Comment:
recent version of opensearch allows disabling _source field: https://docs.opensearch.org/latest/mappings/metadata-fields/source/#disabling-the-field, more so to reduce index size when vector embedding is also stored along. @ryerraguntla Do you think it will affect the functionality here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on PR #3515:
URL: https://github.com/apache/iggy/pull/3515#issuecomment-4884867037
Tests Unrelated to PR are failing. Need to find the root cause and solve the flakiness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on PR #3515:
URL: https://github.com/apache/iggy/pull/3515#issuecomment-4888324281
@hubcio @kriti-sc - The preChecks are failing (flaky) with java SDK test, which is completely unrelated to the PR related changes. [Pre-merge / Java • test / run (pull_request)](https://github.com/apache/iggy/actions/runs/28737059115/job/85213039410?pr=3515) ---AsyncConsumerGroupsTest , shouldHandleConcurrentJoinAndLeaveOperations . Since it is not related to this PR, can I raise an Issue and a new PR for fixing this failure. Please let me know if there are any other ways of handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #3515:
URL: https://github.com/apache/iggy/pull/3515#discussion_r3508951798
##########
core/connectors/sources/opensearch_source/README.md:
##########
@@ -0,0 +1,296 @@
+# OpenSearch Source Connector
+
+Polls documents from an OpenSearch index and publishes them to Iggy streams as JSON
+messages. Incremental progress is tracked with OpenSearch `search_after` pagination
+on `(timestamp_field, _id)`.
+
+## Architecture
+
+The connector is a cdylib source plugin loaded by the Iggy connectors runtime via FFI.
+
+| Layer | Crate / binary | Role |
+| ----- | -------------- | ---- |
+| Plugin | `iggy_connector_opensearch_source` | Implements `Source` trait; talks to OpenSearch |
+| SDK | `iggy_connector_sdk` | `Source`, `ProducedMessage`, `ConnectorState`, `source_connector!` macro |
+| Runtime | `iggy-connectors` | Loads `.dylib`, calls `open` / `poll` / `close`, publishes to Iggy, saves `ConnectorState` |
+| Server | `iggy-server` | Receives messages on configured streams/topics |
+
+The connector is read-only. It does not write to OpenSearch.
+
+## Configuration
+
+```toml
+type = "source"
+key = "opensearch"
+enabled = true
+version = 0
+name = "OpenSearch source"
+path = "target/release/libiggy_connector_opensearch_source"
+plugin_config_format = "json"
+
+[[streams]]
+stream = "opensearch_stream"
+topic = "documents"
+schema = "json"
+batch_length = 100
+linger_time = "5ms"
+
+[plugin_config]
+url = "http://localhost:9200"
+index = "logs-*"
+polling_interval = "10s"
+batch_size = 100
+timestamp_field = "@timestamp"
+query = { match_all = {} }
+```
+
+### Required fields
+
+| Field | Type | Description |
+| ----- | ---- | ----------- |
+| `url` | `String` | OpenSearch HTTP base URL |
+| `index` | `String` | Index name or pattern |
+| `timestamp_field` | `String` | Document field used for sort order and cursor; must exist on every document |
+
+### Optional fields
+
+| Field | Type | Default | Description |
+| ----- | ---- | ------- | ----------- |
+| `polling_interval` | `String` | `"10s"` | Delay after each completed poll cycle (humantime format). First poll runs immediately. |
+| `batch_size` | `usize` | `100` | Documents per search request (minimum `1`) |
+| `query` | JSON object | `{"match_all": {}}` | OpenSearch query DSL; applied on every poll |
+| `username` / `password` | `String` | none | HTTP basic authentication |
+| `verbose_logging` | `bool` | `false` | Log per-poll batch counts at `info!` instead of `debug!` |
+| `max_retries` | `u32` | `3` | Total HTTP attempts per search during `poll()` |
+| `retry_delay` | `String` | `"1s"` | Base backoff between HTTP retries |
+| `retry_max_delay` | `String` | `"30s"` | Maximum backoff between HTTP retries |
+| `max_open_retries` | `u32` | `5` | Total attempts for index-exists check during `open()` |
+| `open_retry_max_delay` | `String` | `"30s"` | Maximum backoff during `open()` probes |
+| `circuit_breaker_threshold` | `u32` | `5` | Consecutive poll failures before circuit opens |
+| `circuit_breaker_cool_down` | `String` | `"60s"` | Duration to skip polls when circuit is open |
+
+### File-backed state (optional)
+
+Runtime state is always returned from `poll()` and persisted by the connectors
+runtime. To additionally mirror state to JSON files on disk:
+
+```toml
+[plugin_config.state]
+enabled = true
+storage_type = "file"
+storage_config = { base_path = "./connector_states" }
+state_id = "opensearch_logs_connector"
+```
+
+Only `storage_type = "file"` is implemented. See [State and persistence](#state-and-persistence).
+
+## How it works
+
+### Poll cycle
+
+Each call to `poll()`:
+
+1. Issues `POST /{index}/_search` with the query below.
+2. Maps each hit's `_source` to a JSON `ProducedMessage`.
+3. Updates the `search_after` cursor to the sort tuple of the last hit with a valid sort tuple.
+4. Returns `ProducedMessages` containing the messages and a serialized `ConnectorState`.
+5. Sleeps `polling_interval` before returning.
+
+The runtime persists `ConnectorState` (msgpack) after each successful `poll()` return.
+
+### Search request
+
+```json
+{
+ "query": "<config.query or match_all>",
+ "size": "<batch_size>",
+ "sort": [
+ { "<timestamp_field>": { "order": "asc" } },
+ { "_id": { "order": "asc" } }
+ ],
+ "search_after": ["<omitted on first poll; previous batch's sort tuple thereafter>"]
+}
+```
+
+Two sort keys give stable order when timestamps collide. `_id` is the tiebreaker.
+
+### Per-hit processing
+
+For each hit in the response:
+
+1. **Missing sort tuple** — skip with `warn!` for that hit; if no hit in the batch has a valid sort tuple, the poll fails with `Error::Storage`.
+2. **Missing `_source`** — skip with `warn!` (not published); cursor still advances to that hit's sort position.
+3. **Both present** — serialize `_source` as JSON payload.
+
+The cursor (`search_after`) advances for any hit with a valid sort tuple, including hits skipped for missing `_source`. An empty batch leaves the cursor unchanged.
+
+### Timestamp parsing
+
+The `timestamp_field` value in `_source` is parsed to populate `last_poll_timestamp` (informational only; does not affect pagination).
+
+| `_source` value | Parsing |
+| --------------- | ------- |
+| RFC 3339 string | `DateTime::parse_from_rfc3339` |
+| Integer `> 1e12` | Epoch milliseconds |
+| Integer `≤ 1e12` | Epoch seconds |
+| Other | Ignored; document still published |
+
+## State and persistence
+
+### Internal state fields
+
+| Field | Purpose |
+| ----- | ------- |
+| `search_after` | `Option<Vec<Value>>` — OpenSearch sort tuple from last published hit; authoritative resume cursor |
+| `last_poll_timestamp` | `Option<DateTime<Utc>>` — timestamp of last processed document; informational |
+| `total_documents_published` | Cumulative documents emitted to Iggy |
+| `poll_count` | Total search requests executed (successful + empty) |
+| `error_count` / `last_error` | Search failure tracking |
+| `processing_stats` | Bytes processed, empty/successful poll counts, avg latency |
+
+**Invariant:** `search_after` is the authoritative resume cursor. `last_poll_timestamp` is
+informational only and does not affect pagination.
+
+### Dual persistence
+
+| Mechanism | Format | When written | When read | Failure mode |
+| --------- | ------ | ------------ | --------- | ------------ |
+| Runtime `ConnectorState` | MessagePack | Every `poll()` return | `new(id, config, Some(state))` | Corrupt → `open()` fails with `InitError` |
+| File `SourceState` | JSON | `close()` if `state.enabled` and connector opened successfully | `open()` if `state.enabled` and no runtime state present | Load failure → `open()` fails with `InitError` |
+
+File path: `{base_path}/{state_id}.json`; defaults: `base_path = "./connector_states"`,
+`state_id = "opensearch_source_{id}"`.
+
+Runtime `ConnectorState` is authoritative. When valid runtime state is restored on
+startup, file state is not loaded. File mirror is written atomically (write-tmp →
+fdatasync → rename → dir-fsync) on `close()`.
+
+## Initial load and tuning
+
+### Cursor behavior by phase
+
+| Phase | Cursor behavior |
+| ----- | --------------- |
+| Fresh start (no state) | No `search_after` — reads from start of sort order |
+| Steady state | `search_after` advances — only documents after cursor returned |
+| Restart with saved state | Cursor restored from `ConnectorState`; resumes without re-reading |
+
+There is no separate initial-load code path. Every poll uses the same logic.
+
+### Throughput
+
+With defaults (`batch_size = 100`, `polling_interval = "10s"`):
+
+```text
+100 docs / 10s ≈ 10 docs/sec
+10,000,000 docs ≈ ~11.5 days to catch up
+```
+
+Aggressive config for large initial loads:
+
+```toml
+[plugin_config]
+polling_interval = "100ms"
+batch_size = 5000
+timestamp_field = "@timestamp"
+```
+
+Optional time-window queries for manual partitioning:
+
+```toml
+[plugin_config]
+query = { "range" = { "@timestamp" = { "gte" = "2024-01-01", "lt" = "2024-02-01" } } }
+```
+
+Requirements for correct operation:
+
+- `timestamp_field` present on every document.
+- Index mapping has a date-type field for `timestamp_field`.
+- `_source` enabled in the index mapping (see Limitations).
+
+## Error handling
+
+| Error variant | When raised |
+| ------------- | ----------- |
+| `InitError` | Corrupt runtime state; missing index at `open()`; file state load failure |
+| `InvalidConfigValue` | Missing `timestamp_field`; `batch_size = 0`; unsupported `storage_type` |
+| `Storage` | Network or HTTP errors; client not initialized at `poll()` |
+| `Serialization` | JSON / MessagePack failures |
+
+## Resilience
+
+### Retry policy
+
+| Category | Conditions |
+| -------- | ---------- |
+| Transient (retry) | Network errors; HTTP `429`; HTTP `5xx`; honors `Retry-After` header on `429` |
+| Permanent (no retry) | `400`, `401`, `403`, `404`; malformed responses; search DSL errors |
+
+### Circuit breaker
+
+When open, `poll()` skips the search, logs a warning, sleeps `polling_interval`, and returns an empty batch with no state update - cursor does not advance. Consecutive failures after retries exhausted increment the breaker; a successful search resets it.
+
+### Delivery semantics
+
+**At-least-once toward Iggy.** The in-memory cursor advances in `finalize_poll()` before the runtime persists `ConnectorState`. A crash after cursor advance but before the runtime save re-emits the last batch on restart.
Review Comment:
"at-least-once" only holds across a crash. on an in-process `send()` failure the runtime logs the error and keeps polling without persisting state, but the in-memory cursor already advanced in `finalize_poll()` - the failed batch gets skipped until the next restart, i.e. at-most-once for that batch. worth stating that case here too, since persist-on-send-success is the runtime's contract for all sources.
##########
core/connectors/runtime/example_config/connectors/opensearch_source.toml:
##########
@@ -0,0 +1,63 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+type = "source"
+key = "opensearch"
+enabled = true
+version = 0
+name = "OpenSearch source"
+path = "<BASE_DIR>/target/release/libiggy_connector_opensearch_source"
+plugin_config_format = "json"
+
+[[streams]]
+stream = "opensearch_stream"
+topic = "documents"
+schema = "json"
+batch_length = 100
+linger_time = "5ms"
+
+[plugin_config]
+url = "http://localhost:9200"
+index = "logs-*"
+polling_interval = "10s"
+batch_size = 100
+timestamp_field = "@timestamp"
+# username = "admin"
+# password = "replace_with_secret"
+# verbose_logging = false
+
+# HTTP resilience (defaults shown; see docs/RESILIENCE.md).
Review Comment:
`docs/RESILIENCE.md` doesn't exist anywhere in the repo.
##########
core/connectors/sources/opensearch_source/dependencies.md:
##########
@@ -0,0 +1,46 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# OpenSearch Source Connector — Direct Runtime Dependencies
+
+This file lists every direct (non-dev) dependency declared in
+`core/connectors/sources/opensearch_source/Cargo.toml`, together with
+its workspace-pinned version, license, and the specific role it plays
+in this connector. Transitive dependencies are not listed here; refer
+to `cargo tree -p iggy_connector_opensearch_source` for the full graph.
+
+---
+
+## Runtime dependencies
+
+| Crate | Version (workspace) | License | Role in this connector |
+| --- | --- | --- | --- |
+| `async-trait` | `^0.1.89` | MIT / Apache-2.0 | Proc-macro that enables `async fn` in trait definitions; required by the `Source` trait impl in `lib.rs`. |
+| `dashmap` | `^6.1.0` | MIT | Concurrent hash map; injected into this crate's namespace by the `source_connector!` macro expansion in the SDK. Not used directly in source files. |
+| `humantime` | `^2.3.0` | MIT / Apache-2.0 | Workspace dependency; duration parsing uses `iggy_connector_sdk::retry::parse_duration`. |
Review Comment:
`humantime` isn't in this crate's `[dependencies]` - the row itself admits it's transitive via the SDK's `parse_duration`, contradicting the header's "every direct (non-dev) dependency" claim. rather than fixing the row, suggest deleting this file: it duplicates `Cargo.toml` and drifts silently (this row is the proof). the closest sibling `elasticsearch_source` ships without one (only the influxdb connectors carry it), and license enumeration is already handled by the third-party-licenses CI script.
delete this dependencies.md file.
##########
core/connectors/sources/opensearch_source/README.md:
##########
@@ -0,0 +1,296 @@
+# OpenSearch Source Connector
+
+Polls documents from an OpenSearch index and publishes them to Iggy streams as JSON
+messages. Incremental progress is tracked with OpenSearch `search_after` pagination
+on `(timestamp_field, _id)`.
+
+## Architecture
+
+The connector is a cdylib source plugin loaded by the Iggy connectors runtime via FFI.
+
+| Layer | Crate / binary | Role |
+| ----- | -------------- | ---- |
+| Plugin | `iggy_connector_opensearch_source` | Implements `Source` trait; talks to OpenSearch |
+| SDK | `iggy_connector_sdk` | `Source`, `ProducedMessage`, `ConnectorState`, `source_connector!` macro |
+| Runtime | `iggy-connectors` | Loads `.dylib`, calls `open` / `poll` / `close`, publishes to Iggy, saves `ConnectorState` |
+| Server | `iggy-server` | Receives messages on configured streams/topics |
+
+The connector is read-only. It does not write to OpenSearch.
+
+## Configuration
+
+```toml
+type = "source"
+key = "opensearch"
+enabled = true
+version = 0
+name = "OpenSearch source"
+path = "target/release/libiggy_connector_opensearch_source"
+plugin_config_format = "json"
+
+[[streams]]
+stream = "opensearch_stream"
+topic = "documents"
+schema = "json"
+batch_length = 100
+linger_time = "5ms"
+
+[plugin_config]
+url = "http://localhost:9200"
+index = "logs-*"
+polling_interval = "10s"
+batch_size = 100
+timestamp_field = "@timestamp"
+query = { match_all = {} }
+```
+
+### Required fields
+
+| Field | Type | Description |
+| ----- | ---- | ----------- |
+| `url` | `String` | OpenSearch HTTP base URL |
+| `index` | `String` | Index name or pattern |
+| `timestamp_field` | `String` | Document field used for sort order and cursor; must exist on every document |
+
+### Optional fields
+
+| Field | Type | Default | Description |
+| ----- | ---- | ------- | ----------- |
+| `polling_interval` | `String` | `"10s"` | Delay after each completed poll cycle (humantime format). First poll runs immediately. |
+| `batch_size` | `usize` | `100` | Documents per search request (minimum `1`) |
+| `query` | JSON object | `{"match_all": {}}` | OpenSearch query DSL; applied on every poll |
+| `username` / `password` | `String` | none | HTTP basic authentication |
+| `verbose_logging` | `bool` | `false` | Log per-poll batch counts at `info!` instead of `debug!` |
+| `max_retries` | `u32` | `3` | Total HTTP attempts per search during `poll()` |
+| `retry_delay` | `String` | `"1s"` | Base backoff between HTTP retries |
+| `retry_max_delay` | `String` | `"30s"` | Maximum backoff between HTTP retries |
+| `max_open_retries` | `u32` | `5` | Total attempts for index-exists check during `open()` |
+| `open_retry_max_delay` | `String` | `"30s"` | Maximum backoff during `open()` probes |
+| `circuit_breaker_threshold` | `u32` | `5` | Consecutive poll failures before circuit opens |
+| `circuit_breaker_cool_down` | `String` | `"60s"` | Duration to skip polls when circuit is open |
+
+### File-backed state (optional)
+
+Runtime state is always returned from `poll()` and persisted by the connectors
+runtime. To additionally mirror state to JSON files on disk:
+
+```toml
+[plugin_config.state]
+enabled = true
+storage_type = "file"
+storage_config = { base_path = "./connector_states" }
+state_id = "opensearch_logs_connector"
+```
+
+Only `storage_type = "file"` is implemented. See [State and persistence](#state-and-persistence).
+
+## How it works
+
+### Poll cycle
+
+Each call to `poll()`:
+
+1. Issues `POST /{index}/_search` with the query below.
+2. Maps each hit's `_source` to a JSON `ProducedMessage`.
+3. Updates the `search_after` cursor to the sort tuple of the last hit with a valid sort tuple.
+4. Returns `ProducedMessages` containing the messages and a serialized `ConnectorState`.
+5. Sleeps `polling_interval` before returning.
+
+The runtime persists `ConnectorState` (msgpack) after each successful `poll()` return.
+
+### Search request
+
+```json
+{
+ "query": "<config.query or match_all>",
+ "size": "<batch_size>",
+ "sort": [
+ { "<timestamp_field>": { "order": "asc" } },
+ { "_id": { "order": "asc" } }
+ ],
+ "search_after": ["<omitted on first poll; previous batch's sort tuple thereafter>"]
+}
+```
+
+Two sort keys give stable order when timestamps collide. `_id` is the tiebreaker.
+
+### Per-hit processing
+
+For each hit in the response:
+
+1. **Missing sort tuple** — skip with `warn!` for that hit; if no hit in the batch has a valid sort tuple, the poll fails with `Error::Storage`.
+2. **Missing `_source`** — skip with `warn!` (not published); cursor still advances to that hit's sort position.
+3. **Both present** — serialize `_source` as JSON payload.
+
+The cursor (`search_after`) advances for any hit with a valid sort tuple, including hits skipped for missing `_source`. An empty batch leaves the cursor unchanged.
+
+### Timestamp parsing
+
+The `timestamp_field` value in `_source` is parsed to populate `last_poll_timestamp` (informational only; does not affect pagination).
+
+| `_source` value | Parsing |
+| --------------- | ------- |
+| RFC 3339 string | `DateTime::parse_from_rfc3339` |
+| Integer `> 1e12` | Epoch milliseconds |
+| Integer `≤ 1e12` | Epoch seconds |
+| Other | Ignored; document still published |
+
+## State and persistence
+
+### Internal state fields
+
+| Field | Purpose |
+| ----- | ------- |
+| `search_after` | `Option<Vec<Value>>` — OpenSearch sort tuple from last published hit; authoritative resume cursor |
+| `last_poll_timestamp` | `Option<DateTime<Utc>>` — timestamp of last processed document; informational |
+| `total_documents_published` | Cumulative documents emitted to Iggy |
+| `poll_count` | Total search requests executed (successful + empty) |
+| `error_count` / `last_error` | Search failure tracking |
+| `processing_stats` | Bytes processed, empty/successful poll counts, avg latency |
+
+**Invariant:** `search_after` is the authoritative resume cursor. `last_poll_timestamp` is
+informational only and does not affect pagination.
+
+### Dual persistence
+
+| Mechanism | Format | When written | When read | Failure mode |
+| --------- | ------ | ------------ | --------- | ------------ |
+| Runtime `ConnectorState` | MessagePack | Every `poll()` return | `new(id, config, Some(state))` | Corrupt → `open()` fails with `InitError` |
+| File `SourceState` | JSON | `close()` if `state.enabled` and connector opened successfully | `open()` if `state.enabled` and no runtime state present | Load failure → `open()` fails with `InitError` |
+
+File path: `{base_path}/{state_id}.json`; defaults: `base_path = "./connector_states"`,
+`state_id = "opensearch_source_{id}"`.
+
+Runtime `ConnectorState` is authoritative. When valid runtime state is restored on
+startup, file state is not loaded. File mirror is written atomically (write-tmp →
+fdatasync → rename → dir-fsync) on `close()`.
+
+## Initial load and tuning
+
+### Cursor behavior by phase
+
+| Phase | Cursor behavior |
+| ----- | --------------- |
+| Fresh start (no state) | No `search_after` — reads from start of sort order |
+| Steady state | `search_after` advances — only documents after cursor returned |
+| Restart with saved state | Cursor restored from `ConnectorState`; resumes without re-reading |
+
+There is no separate initial-load code path. Every poll uses the same logic.
+
+### Throughput
+
+With defaults (`batch_size = 100`, `polling_interval = "10s"`):
+
+```text
+100 docs / 10s ≈ 10 docs/sec
+10,000,000 docs ≈ ~11.5 days to catch up
+```
+
+Aggressive config for large initial loads:
+
+```toml
+[plugin_config]
+polling_interval = "100ms"
+batch_size = 5000
+timestamp_field = "@timestamp"
+```
+
+Optional time-window queries for manual partitioning:
+
+```toml
+[plugin_config]
+query = { "range" = { "@timestamp" = { "gte" = "2024-01-01", "lt" = "2024-02-01" } } }
+```
+
+Requirements for correct operation:
+
+- `timestamp_field` present on every document.
+- Index mapping has a date-type field for `timestamp_field`.
+- `_source` enabled in the index mapping (see Limitations).
+
+## Error handling
+
+| Error variant | When raised |
+| ------------- | ----------- |
+| `InitError` | Corrupt runtime state; missing index at `open()`; file state load failure |
+| `InvalidConfigValue` | Missing `timestamp_field`; `batch_size = 0`; unsupported `storage_type` |
+| `Storage` | Network or HTTP errors; client not initialized at `poll()` |
+| `Serialization` | JSON / MessagePack failures |
+
+## Resilience
+
+### Retry policy
+
+| Category | Conditions |
+| -------- | ---------- |
+| Transient (retry) | Network errors; HTTP `429`; HTTP `5xx`; honors `Retry-After` header on `429` |
Review Comment:
code honors `Retry-After` on 5xx as well, not just 429 - `sleep_before_retry` receives the header for any transient status and `is_transient_status` covers 429 + 5xx. correct behavior (503 with `Retry-After` is legit per RFC 7231), just undersold here.
##########
core/connectors/sources/opensearch_source/src/lib.rs:
##########
@@ -0,0 +1,1136 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::retry::{CircuitBreaker, parse_duration};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use opensearch::{
+ OpenSearch, SearchParts,
+ auth::Credentials,
+ http::{Url, transport::TransportBuilder},
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use serde_json::{Value, json};
+use std::sync::Arc;
+use std::time::Duration;
+use tokio::{sync::Mutex, time::sleep};
+use tracing::{debug, error, info, warn};
+
+mod retry;
+mod state_manager;
+use crate::retry::{
+ DEFAULT_CB_COOL_DOWN, DEFAULT_CB_THRESHOLD, DEFAULT_MAX_OPEN_RETRIES, DEFAULT_MAX_RETRIES,
+ DEFAULT_OPEN_RETRY_MAX_DELAY, DEFAULT_RETRY_DELAY, DEFAULT_RETRY_MAX_DELAY, RetryBackoff,
+ is_transient_status, normalized_max_attempts, sleep_before_retry,
+};
+use crate::state_manager::{SOURCE_STATE_VERSION, SourceState, validate_state_storage_config};
+
+source_connector!(OpenSearchSource);
+
+const CONNECTOR_NAME: &str = "OpenSearch source";
+const DEFAULT_POLLING_INTERVAL: &str = "10s";
+const DEFAULT_BATCH_SIZE: usize = 100;
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct State {
+ #[serde(default)]
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ #[serde(default, alias = "total_documents_fetched")]
+ total_documents_published: usize,
+ #[serde(default)]
+ poll_count: usize,
+ /// OpenSearch `search_after` tuple from the last hit in the previous batch.
+ #[serde(default)]
+ search_after: Option<Vec<Value>>,
+ #[serde(default)]
+ error_count: usize,
+ #[serde(default)]
+ last_error: Option<String>,
+ #[serde(default)]
+ processing_stats: ProcessingStats,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct ProcessingStats {
+ #[serde(default)]
+ total_bytes_processed: u64,
+ /// Running cumulative average over the connector's lifetime, persisted and accumulated
+ /// across restarts. Reflects long-term throughput baseline, not session-only average.
+ #[serde(default)]
+ avg_batch_processing_time_ms: f64,
+ #[serde(default)]
+ last_successful_poll: Option<DateTime<Utc>>,
+ #[serde(default)]
+ empty_polls_count: usize,
+ #[serde(default)]
+ successful_polls_count: usize,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct StateConfig {
+ #[serde(default)]
+ pub enabled: bool,
+ pub storage_type: Option<String>,
+ pub storage_config: Option<Value>,
+ pub state_id: Option<String>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct OpenSearchSourceConfig {
+ pub url: String,
+ pub index: String,
+ pub username: Option<String>,
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_optional_secret")]
+ pub password: Option<SecretString>,
+ pub query: Option<Value>,
+ pub polling_interval: Option<String>,
+ pub batch_size: Option<usize>,
+ pub timestamp_field: Option<String>,
+ #[serde(default)]
+ pub verbose_logging: bool,
+ pub state: Option<StateConfig>,
+ /// Total poll-phase attempt count (not retry count); 1 = no retries, 3 = 2 retries.
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+ pub retry_max_delay: Option<String>,
+ /// Total open-phase attempt count (not retry count); 1 = no retries, 5 = 4 retries.
+ pub max_open_retries: Option<u32>,
+ pub open_retry_max_delay: Option<String>,
+ pub circuit_breaker_threshold: Option<u32>,
+ pub circuit_breaker_cool_down: Option<String>,
+}
+
+#[derive(Debug)]
+pub struct OpenSearchSource {
+ id: u32,
+ config: OpenSearchSourceConfig,
+ client: Option<OpenSearch>,
+ polling_interval: Duration,
+ /// Pre-built query body (query + size + sort). Set once in `open()` after config validation.
+ /// `None` before `open()` and after a failed `open()`.
+ search_body_base: Option<Value>,
+ verbose: bool,
+ max_retries: u32,
+ retry_delay: Duration,
+ retry_max_delay: Duration,
+ max_open_retries: u32,
+ open_retry_max_delay: Duration,
+ circuit_breaker: Arc<CircuitBreaker>,
+ state: Mutex<State>,
+ /// `Some(cause)` when runtime state restore was rejected; `None` means restore succeeded.
+ state_restore_error: Option<String>,
+ /// True when `new()` restored a valid runtime `ConnectorState`. File mirror must not override it.
+ runtime_state_restored: bool,
+}
+
+struct SearchOutcome {
+ messages: Vec<ProducedMessage>,
+ search_after: Option<Vec<Value>>,
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ batch_bytes: u64,
+}
+
+impl OpenSearchSource {
+ pub fn new(id: u32, config: OpenSearchSourceConfig, state: Option<ConnectorState>) -> Self {
+ let polling_interval =
+ parse_duration(config.polling_interval.as_deref(), DEFAULT_POLLING_INTERVAL);
+ let verbose = config.verbose_logging;
+ let (restored_state, state_restore_error, runtime_state_restored) =
+ restore_state(id, state);
+
+ let cb_threshold = config
+ .circuit_breaker_threshold
+ .unwrap_or(DEFAULT_CB_THRESHOLD);
+ let cb_cool_down = parse_duration(
+ config.circuit_breaker_cool_down.as_deref(),
+ DEFAULT_CB_COOL_DOWN,
+ );
+ let circuit_breaker = Arc::new(CircuitBreaker::new(cb_threshold, cb_cool_down));
+ let max_retries =
+ normalized_max_attempts(config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES));
+ let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+ let retry_max_delay =
+ parse_duration(config.retry_max_delay.as_deref(), DEFAULT_RETRY_MAX_DELAY);
+ let max_open_retries =
+ normalized_max_attempts(config.max_open_retries.unwrap_or(DEFAULT_MAX_OPEN_RETRIES));
+ let open_retry_max_delay = parse_duration(
+ config.open_retry_max_delay.as_deref(),
+ DEFAULT_OPEN_RETRY_MAX_DELAY,
+ );
+
+ OpenSearchSource {
+ id,
+ config,
+ max_retries,
+ retry_delay,
+ retry_max_delay,
+ max_open_retries,
+ open_retry_max_delay,
+ circuit_breaker,
+ client: None,
+ polling_interval,
+ search_body_base: None,
+ verbose,
+ state: Mutex::new(restored_state),
+ state_restore_error,
+ runtime_state_restored,
+ }
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn batch_size(&self) -> usize {
+ self.config.batch_size.unwrap_or(DEFAULT_BATCH_SIZE)
+ }
+
+ fn timestamp_field(&self) -> &str {
+ self.config
+ .timestamp_field
+ .as_deref()
+ .expect("timestamp_field validated at open()")
+ }
+
+ fn get_state_id(&self) -> String {
+ self.config
+ .state
+ .as_ref()
+ .and_then(|s| s.state_id.clone())
+ .unwrap_or_else(|| format!("opensearch_source_{}", self.id))
+ }
+
+ pub(crate) async fn internal_state_to_source_state(&self) -> Result<SourceState, Error> {
+ let state = self.state.lock().await;
+ let data = serde_json::to_value(&*state).map_err(|error| {
+ Error::Serialization(format!("Failed to serialize connector state: {error}"))
+ })?;
+
+ Ok(SourceState {
+ id: self.get_state_id(),
+ last_updated: Utc::now(),
+ version: SOURCE_STATE_VERSION,
+ data,
+ metadata: Some(json!({
+ "connector_type": "opensearch_source",
+ "connector_id": self.id,
+ "index": self.config.index,
+ "url": self.config.url,
+ })),
+ })
+ }
+
+ pub(crate) async fn source_state_to_internal_state(
+ &mut self,
+ source_state: SourceState,
+ ) -> Result<(), Error> {
+ if source_state.version != SOURCE_STATE_VERSION {
+ return Err(Error::Serialization(format!(
+ "unsupported file state version {}, expected {SOURCE_STATE_VERSION}",
+ source_state.version
+ )));
+ }
+
+ let restored: State = serde_json::from_value(source_state.data).map_err(|error| {
+ Error::Serialization(format!("Failed to deserialize connector state: {error}"))
+ })?;
+
+ let mut state = self.state.lock().await;
+ *state = restored;
+ Ok(())
+ }
+
+ async fn create_client(&self) -> Result<OpenSearch, Error> {
+ let url = Url::parse(&self.config.url).map_err(|error| {
+ Error::InvalidConfigValue(format!("Invalid OpenSearch URL: {error}"))
+ })?;
+
+ let conn_pool = opensearch::http::transport::SingleNodeConnectionPool::new(url);
+ let mut transport_builder = TransportBuilder::new(conn_pool);
+
+ if let (Some(username), Some(password)) = (&self.config.username, &self.config.password) {
+ let credentials =
+ Credentials::Basic(username.clone(), password.expose_secret().to_string());
+ transport_builder = transport_builder.auth(credentials);
+ }
+
+ let transport = transport_builder
+ .build()
+ .map_err(|error| Error::InitError(format!("Failed to build transport: {error}")))?;
+
+ Ok(OpenSearch::new(transport))
+ }
+
+ async fn check_index_exists_with_retry(&self, client: &OpenSearch) -> Result<(), Error> {
+ let max_attempts = self.max_open_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let response = match client
+ .indices()
+ .exists(opensearch::indices::IndicesExistsParts::Index(&[&self
+ .config
+ .index]))
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ // retry_delay is shared with the poll phase (no separate open-phase
+ // base delay); open_retry_max_delay caps the growth independently.
+ // Same pattern as the InfluxDB source connector.
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::InitError(format!(
+ "Failed to check index existence: {error}"
+ )));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(());
+ }
+
+ let status = response.status_code().as_u16();
+ if status == 404 {
+ return Err(Error::InitError(format!(
+ "Index '{}' does not exist or is not accessible",
+ self.config.index
+ )));
+ }
+
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::InitError(format!(
+ "Index '{}' check failed: HTTP {status}: {error_text}",
+ self.config.index
+ )));
+ }
+ }
+
+ async fn send_search_with_retry(
+ &self,
+ client: &OpenSearch,
+ mut search_body: Value,
+ ) -> Result<opensearch::http::response::Response, Error> {
+ let max_attempts = self.max_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let body = if attempt < max_attempts {
+ search_body.clone()
+ } else {
+ std::mem::take(&mut search_body)
+ };
+ let response = match client
+ .search(SearchParts::Index(&[&self.config.index]))
+ .body(body)
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::Storage(format!("Failed to execute search: {error}")));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(response);
+ }
+
+ let status = response.status_code().as_u16();
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::Storage(format!(
+ "Search request failed: {error_text}"
+ )));
+ }
+ }
+
+ async fn search_documents(&self, client: &OpenSearch) -> Result<SearchOutcome, Error> {
+ let state = self.state.lock().await;
+ let search_after = state.search_after.clone();
+ drop(state);
+
+ let timestamp_field = self.timestamp_field();
+
+ let mut search_body = self
+ .search_body_base
+ .as_ref()
+ .ok_or_else(|| {
+ Error::Connection("connector not initialized; call open() first".to_string())
+ })?
+ .clone();
+
+ if let Some(cursor) = search_after {
+ search_body["search_after"] = json!(cursor);
+ }
+
+ let response = self.send_search_with_retry(client, search_body).await?;
+
+ let mut response_body: Value = response
+ .json()
+ .await
+ .map_err(|e| Error::Storage(format!("Failed to parse search response: {e}")))?;
+
+ let hits: Vec<Value> = response_body
+ .get_mut("hits")
+ .and_then(|h| h.get_mut("hits"))
+ .and_then(|arr| arr.as_array_mut())
+ .map(std::mem::take)
+ .unwrap_or_default();
+
+ let mut messages = Vec::with_capacity(hits.len());
+ let mut batch_bytes = 0u64;
+ let mut last_sort = None;
+ let mut last_poll_timestamp = None;
+
+ for hit in &hits {
+ let Some(sort) = hit
+ .get("sort")
+ .and_then(|s| s.as_array())
+ .filter(|a| !a.is_empty())
+ else {
+ warn!(
+ connector_id = self.id,
+ hit_id = hit.get("_id").and_then(|value| value.as_str()),
+ "Skipping OpenSearch hit without sort tuple; document will not be published"
+ );
+ continue;
+ };
+
+ last_sort = Some(sort);
+
+ let Some(source) = hit.get("_source") else {
+ warn!(
+ connector_id = self.id,
+ hit_id = hit.get("_id").and_then(|v| v.as_str()),
+ "Skipping OpenSearch hit without _source; document will not be published"
+ );
+ continue;
+ };
+
+ if let Some(timestamp_value) = source.get(timestamp_field)
+ && let Some(timestamp_utc) = parse_document_timestamp(timestamp_value)
+ {
+ last_poll_timestamp = Some(timestamp_utc);
+ }
+
+ let payload = serde_json::to_vec(source)
+ .map_err(|e| Error::Serialization(format!("Failed to serialize document: {e}")))?;
+ batch_bytes += payload.len() as u64;
+
+ messages.push(ProducedMessage {
+ id: None,
+ headers: None,
+ checksum: None,
+ timestamp: None,
+ origin_timestamp: None,
+ payload,
+ });
+ }
+
+ if !hits.is_empty() && last_sort.is_none() {
+ return Err(Error::Storage(format!(
+ "OpenSearch returned {} hit(s) but none had a sort tuple; \
+ index may be missing the sort field or using an incompatible mapping",
+ hits.len()
+ )));
+ }
+
+ // Guard: cursor would advance past the entire batch but nothing would be published.
+ // This happens when _source is disabled on the index (all sort-bearing hits lack _source).
+ if !hits.is_empty() && messages.is_empty() && last_sort.is_some() {
+ return Err(Error::Storage(format!(
+ "OpenSearch returned {} hit(s) with valid sort tuples but all were missing \
+ _source; index may have _source disabled. Refusing to advance cursor \
+ without publishing any messages.",
+ hits.len()
+ )));
+ }
+
+ Ok(SearchOutcome {
+ messages,
+ search_after: last_sort.map(ToOwned::to_owned),
+ last_poll_timestamp,
+ batch_bytes,
+ })
+ }
+
+ async fn finalize_poll(
+ &self,
+ outcome: SearchOutcome,
+ processing_time_ms: f64,
+ ) -> (Vec<ProducedMessage>, Option<ConnectorState>) {
+ let mut state = self.state.lock().await;
+ state.total_documents_published += outcome.messages.len();
+ state.poll_count += 1;
+ if let Some(cursor) = outcome.search_after {
+ state.search_after = Some(cursor);
+ }
+ if let Some(timestamp) = outcome.last_poll_timestamp {
+ state.last_poll_timestamp = Some(timestamp);
+ }
+ state.processing_stats.total_bytes_processed += outcome.batch_bytes;
+
+ if outcome.messages.is_empty() {
+ state.processing_stats.empty_polls_count += 1;
+ } else {
+ state.processing_stats.successful_polls_count += 1;
+ state.processing_stats.last_successful_poll = Some(Utc::now());
+ }
+
+ let total_polls = state.processing_stats.successful_polls_count
+ + state.processing_stats.empty_polls_count;
+ // total_polls >= 1 because one of the two counters was just incremented above,
+ // but use saturating_sub to make the invariant machine-checked.
+ state.processing_stats.avg_batch_processing_time_ms =
+ (state.processing_stats.avg_batch_processing_time_ms
+ * total_polls.saturating_sub(1) as f64
+ + processing_time_ms)
+ / total_polls as f64;
+
+ let produced_count = outcome.messages.len();
+ let total_documents_published = state.total_documents_published;
+ let state_snapshot = state.clone();
+ let messages = outcome.messages;
+ drop(state);
+ let persisted_state = self.serialize_state(&state_snapshot);
+
+ if self.verbose {
+ info!(
+ "OpenSearch source connector ID: {} produced {produced_count} messages. \
+ Total published: {total_documents_published}",
+ self.id
+ );
+ } else {
+ debug!(
+ "OpenSearch source connector ID: {} produced {produced_count} messages. \
+ Total published: {total_documents_published}",
+ self.id
+ );
+ }
+
+ (messages, persisted_state)
+ }
+
+ #[cfg(test)]
+ fn client_initialized(&self) -> bool {
+ self.client.is_some()
+ }
+
+ #[cfg(test)]
+ async fn test_metrics(&self) -> (usize, usize, usize, usize) {
+ let state = self.state.lock().await;
+ (
+ state.total_documents_published,
+ state.poll_count,
+ state.error_count,
+ state.processing_stats.empty_polls_count,
+ )
+ }
+
+ #[cfg(test)]
+ async fn test_search_after(&self) -> Option<Vec<Value>> {
+ self.state.lock().await.search_after.clone()
+ }
+
+ #[cfg(test)]
+ async fn test_last_poll_timestamp(&self) -> Option<DateTime<Utc>> {
+ self.state.lock().await.last_poll_timestamp
+ }
+
+ async fn handle_poll_error(&self, error: Error) -> Result<ProducedMessages, Error> {
+ self.circuit_breaker.record_failure().await;
+ let mut state = self.state.lock().await;
+ // error_count and last_error accumulate in State and are captured by the next
+ // finalize_poll call (success path). Both runtime ConnectorState and file state
+ // preserve these values across restarts.
+ state.error_count += 1;
+ state.last_error = Some(error.to_string());
+ drop(state);
+ error!(
+ "{CONNECTOR_NAME} connector ID: {} poll failed: {error}",
+ self.id
+ );
+ sleep(self.polling_interval).await;
+ Err(error)
+ }
+}
+
+fn restore_state(id: u32, state: Option<ConnectorState>) -> (State, Option<String>, bool) {
+ let Some(connector_state) = state else {
+ return (State::default(), None, false);
+ };
+
+ match connector_state.deserialize::<State>(CONNECTOR_NAME, id) {
+ Some(restored) => {
+ info!(
+ "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \
+ Documents published: {}, poll count: {}",
+ restored.total_documents_published, restored.poll_count
+ );
+ (restored, None, true)
+ }
+ None => {
+ let cause = "persisted state exists but could not be deserialized. \
+ Refusing to start to prevent silent cursor reset."
+ .to_string();
+ error!("{CONNECTOR_NAME} ID {id}: {cause}");
+ (State::default(), Some(cause), false)
+ }
+ }
+}
+
+fn validate_open_config(config: &OpenSearchSourceConfig) -> Result<(), Error> {
+ if config.timestamp_field.as_deref().is_none_or(str::is_empty) {
+ return Err(Error::InvalidConfigValue(
+ "timestamp_field is required for incremental OpenSearch polling".to_string(),
+ ));
+ }
+
+ if matches!(config.batch_size, Some(0)) {
+ return Err(Error::InvalidConfigValue(
+ "batch_size must be at least 1".to_string(),
+ ));
+ }
+
+ if let Some(state) = &config.state
+ && state.enabled
+ {
+ validate_state_storage_config(state)?;
+ }
+
+ Ok(())
+}
+
+fn parse_document_timestamp(value: &Value) -> Option<DateTime<Utc>> {
+ match value {
+ Value::String(text) => DateTime::parse_from_rfc3339(text)
+ .ok()
+ .map(|timestamp| timestamp.with_timezone(&Utc)),
+ Value::Number(number) => {
+ let raw = number.as_i64()?;
+ // Values above 1e12 are already milliseconds (Unix epoch seconds won't reach
+ // 1e12 until year 33658). Values at or below are treated as seconds and
+ // multiplied by 1000.
+ let millis = if !(-1_000_000_000_000..=1_000_000_000_000).contains(&raw) {
+ raw
+ } else {
+ raw.saturating_mul(1_000)
+ };
+ DateTime::from_timestamp_millis(millis).map(|timestamp| timestamp.with_timezone(&Utc))
+ }
+ _ => None,
+ }
+}
+
+#[async_trait]
+impl Source for OpenSearchSource {
+ async fn open(&mut self) -> Result<(), Error> {
+ if let Some(ref cause) = self.state_restore_error {
+ return Err(Error::InitError(format!("state restore failed: {cause}")));
+ }
+
+ validate_open_config(&self.config)?;
+
+ let timestamp_field = self.timestamp_field();
+ let batch_size = self.batch_size();
+ self.search_body_base = Some(json!({
+ "query": self.config.query.clone().unwrap_or_else(|| json!({ "match_all": {} })),
+ "size": batch_size,
+ "sort": [
+ { timestamp_field: { "order": "asc" } },
+ { "_id": { "order": "asc" } }
+ ]
+ }));
+
+ info!(
+ "Opening OpenSearch source connector with ID: {} for URL: {}, index: {}",
+ self.id, self.config.url, self.config.index
+ );
+
+ let client = self.create_client().await?;
+
+ self.check_index_exists_with_retry(&client).await?;
+
+ self.client = Some(client);
+
+ if self
+ .config
+ .state
+ .as_ref()
+ .map(|s| s.enabled)
+ .unwrap_or(false)
+ {
+ if self.runtime_state_restored {
+ info!(
+ "Skipping file state load for OpenSearch source connector with ID: {} \
+ because runtime ConnectorState is authoritative",
+ self.id
+ );
+ } else {
+ self.load_state().await.map_err(|error| {
+ Error::InitError(format!("file state load failed: {error}"))
+ })?;
+ }
+ }
+
+ info!(
+ "Successfully opened OpenSearch source connector with ID: {}",
+ self.id
+ );
+ Ok(())
+ }
+
+ async fn poll(&self) -> Result<ProducedMessages, Error> {
+ if self.circuit_breaker.is_open().await {
+ warn!(
+ "{CONNECTOR_NAME} connector ID: {} — circuit breaker is OPEN. Skipping poll.",
+ self.id
+ );
+ sleep(self.polling_interval).await;
+ return Ok(ProducedMessages {
+ schema: Schema::Json,
+ messages: vec![],
+ state: None,
+ });
+ }
+
+ let start_time = std::time::Instant::now();
+
+ let client = self
+ .client
+ .as_ref()
+ .ok_or_else(|| Error::Connection("OpenSearch client not initialized".to_string()))?;
+
+ match self.search_documents(client).await {
+ Ok(outcome) => {
+ self.circuit_breaker.record_success();
+ let processing_time = start_time.elapsed().as_millis() as f64;
+ let (messages, persisted_state) =
+ self.finalize_poll(outcome, processing_time).await;
+ sleep(self.polling_interval).await;
Review Comment:
the sleep runs after `finalize_poll()` but before the batch is returned, so every fetched batch is held in the plugin for a full `polling_interval` before the runtime can send it. both siblings (`elasticsearch_source`, `influxdb_source`) sleep at the top of `poll()` instead - same pacing, but the data they hand back is fresh. throughput is identical either way; this is purely delivery latency. if you move it to the top, mind that the README documents "first poll runs immediately" - a plain move re-introduces the one-interval wait before the first poll, so either update the README to match sibling behavior or skip the sleep on the first iteration.
##########
core/connectors/sources/opensearch_source/src/state_manager.rs:
##########
@@ -0,0 +1,316 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::{OpenSearchSource, StateConfig};
+use async_trait::async_trait;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::Error;
+use serde::{Deserialize, Serialize};
+use std::io::ErrorKind;
+use std::sync::Arc;
+use tracing::info;
+
+pub(crate) const SOURCE_STATE_VERSION: u32 = 1;
+
+impl OpenSearchSource {
+ pub(super) async fn save_state(&self) -> Result<(), Error> {
+ let Some(state_config) = self.config.state.as_ref().filter(|s| s.enabled) else {
+ return Ok(());
+ };
+ let storage = create_state_storage(state_config)?;
+
+ let source_state = self.internal_state_to_source_state().await?;
+ storage.save_source_state(&source_state).await?;
+
+ info!(
+ "Saved state for OpenSearch source connector with ID: {}",
+ self.id
+ );
+ Ok(())
+ }
+
+ pub(super) async fn load_state(&mut self) -> Result<(), Error> {
+ let Some(state_config) = self.config.state.as_ref().filter(|s| s.enabled) else {
+ return Ok(());
+ };
+ let storage = create_state_storage(state_config)?;
+
+ let state_id = self.get_state_id();
+ if let Some(source_state) = storage.load_source_state(&state_id).await? {
+ self.source_state_to_internal_state(source_state).await?;
+
+ let (last_poll_timestamp, total_documents_published, poll_count) = {
+ let state = self.state.lock().await;
+ (
+ state.last_poll_timestamp,
+ state.total_documents_published,
+ state.poll_count,
+ )
+ };
+ info!(
+ "Loaded state for OpenSearch source connector with ID: {} - last poll: {:?}, total docs: {}, polls: {}",
+ self.id, last_poll_timestamp, total_documents_published, poll_count
+ );
+ } else {
+ info!(
+ "No existing state found for OpenSearch source connector with ID: {}, starting fresh",
+ self.id
+ );
+ }
+
+ Ok(())
+ }
+}
+
+pub(crate) fn validate_state_storage_config(config: &StateConfig) -> Result<(), Error> {
+ match config.storage_type.as_deref() {
+ Some("file") | None => Ok(()),
+ Some(storage_type) => Err(Error::InvalidConfigValue(format!(
+ "state storage_type {storage_type:?} is not supported; only \"file\" is implemented"
+ ))),
+ }
+}
+
+pub(crate) fn create_state_storage(config: &StateConfig) -> Result<Arc<dyn StateStorage>, Error> {
+ validate_state_storage_config(config)?;
Review Comment:
both prod callers reach this after `validate_open_config` already ran `validate_state_storage_config` (`load_state` via `open()`, `save_state` via `close()` which is gated on a successful open), so this re-validation can never fail here. safe to drop.
##########
core/connectors/sources/opensearch_source/src/lib.rs:
##########
@@ -0,0 +1,1136 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::retry::{CircuitBreaker, parse_duration};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use opensearch::{
+ OpenSearch, SearchParts,
+ auth::Credentials,
+ http::{Url, transport::TransportBuilder},
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use serde_json::{Value, json};
+use std::sync::Arc;
+use std::time::Duration;
+use tokio::{sync::Mutex, time::sleep};
+use tracing::{debug, error, info, warn};
+
+mod retry;
+mod state_manager;
+use crate::retry::{
+ DEFAULT_CB_COOL_DOWN, DEFAULT_CB_THRESHOLD, DEFAULT_MAX_OPEN_RETRIES, DEFAULT_MAX_RETRIES,
+ DEFAULT_OPEN_RETRY_MAX_DELAY, DEFAULT_RETRY_DELAY, DEFAULT_RETRY_MAX_DELAY, RetryBackoff,
+ is_transient_status, normalized_max_attempts, sleep_before_retry,
+};
+use crate::state_manager::{SOURCE_STATE_VERSION, SourceState, validate_state_storage_config};
+
+source_connector!(OpenSearchSource);
+
+const CONNECTOR_NAME: &str = "OpenSearch source";
+const DEFAULT_POLLING_INTERVAL: &str = "10s";
+const DEFAULT_BATCH_SIZE: usize = 100;
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct State {
+ #[serde(default)]
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ #[serde(default, alias = "total_documents_fetched")]
+ total_documents_published: usize,
+ #[serde(default)]
+ poll_count: usize,
+ /// OpenSearch `search_after` tuple from the last hit in the previous batch.
+ #[serde(default)]
+ search_after: Option<Vec<Value>>,
+ #[serde(default)]
+ error_count: usize,
+ #[serde(default)]
+ last_error: Option<String>,
+ #[serde(default)]
+ processing_stats: ProcessingStats,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct ProcessingStats {
+ #[serde(default)]
+ total_bytes_processed: u64,
+ /// Running cumulative average over the connector's lifetime, persisted and accumulated
+ /// across restarts. Reflects long-term throughput baseline, not session-only average.
+ #[serde(default)]
+ avg_batch_processing_time_ms: f64,
+ #[serde(default)]
+ last_successful_poll: Option<DateTime<Utc>>,
+ #[serde(default)]
+ empty_polls_count: usize,
+ #[serde(default)]
+ successful_polls_count: usize,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct StateConfig {
+ #[serde(default)]
+ pub enabled: bool,
+ pub storage_type: Option<String>,
+ pub storage_config: Option<Value>,
+ pub state_id: Option<String>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct OpenSearchSourceConfig {
+ pub url: String,
+ pub index: String,
+ pub username: Option<String>,
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_optional_secret")]
+ pub password: Option<SecretString>,
+ pub query: Option<Value>,
+ pub polling_interval: Option<String>,
+ pub batch_size: Option<usize>,
+ pub timestamp_field: Option<String>,
+ #[serde(default)]
+ pub verbose_logging: bool,
+ pub state: Option<StateConfig>,
+ /// Total poll-phase attempt count (not retry count); 1 = no retries, 3 = 2 retries.
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+ pub retry_max_delay: Option<String>,
+ /// Total open-phase attempt count (not retry count); 1 = no retries, 5 = 4 retries.
+ pub max_open_retries: Option<u32>,
+ pub open_retry_max_delay: Option<String>,
+ pub circuit_breaker_threshold: Option<u32>,
+ pub circuit_breaker_cool_down: Option<String>,
+}
+
+#[derive(Debug)]
+pub struct OpenSearchSource {
+ id: u32,
+ config: OpenSearchSourceConfig,
+ client: Option<OpenSearch>,
+ polling_interval: Duration,
+ /// Pre-built query body (query + size + sort). Set once in `open()` after config validation.
+ /// `None` before `open()` and after a failed `open()`.
+ search_body_base: Option<Value>,
Review Comment:
comment isn't quite accurate - `search_body_base` is assigned before `create_client()` / `check_index_exists_with_retry()`, so an `open()` that fails past that point leaves it `Some`. harmless (`poll()` gates on `client` first), but the "after a failed `open()`" promise doesn't hold.
##########
core/connectors/sources/opensearch_source/src/lib.rs:
##########
@@ -0,0 +1,1136 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::retry::{CircuitBreaker, parse_duration};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use opensearch::{
+ OpenSearch, SearchParts,
+ auth::Credentials,
+ http::{Url, transport::TransportBuilder},
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use serde_json::{Value, json};
+use std::sync::Arc;
+use std::time::Duration;
+use tokio::{sync::Mutex, time::sleep};
+use tracing::{debug, error, info, warn};
+
+mod retry;
+mod state_manager;
+use crate::retry::{
+ DEFAULT_CB_COOL_DOWN, DEFAULT_CB_THRESHOLD, DEFAULT_MAX_OPEN_RETRIES, DEFAULT_MAX_RETRIES,
+ DEFAULT_OPEN_RETRY_MAX_DELAY, DEFAULT_RETRY_DELAY, DEFAULT_RETRY_MAX_DELAY, RetryBackoff,
+ is_transient_status, normalized_max_attempts, sleep_before_retry,
+};
+use crate::state_manager::{SOURCE_STATE_VERSION, SourceState, validate_state_storage_config};
+
+source_connector!(OpenSearchSource);
+
+const CONNECTOR_NAME: &str = "OpenSearch source";
+const DEFAULT_POLLING_INTERVAL: &str = "10s";
+const DEFAULT_BATCH_SIZE: usize = 100;
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct State {
+ #[serde(default)]
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ #[serde(default, alias = "total_documents_fetched")]
+ total_documents_published: usize,
+ #[serde(default)]
+ poll_count: usize,
+ /// OpenSearch `search_after` tuple from the last hit in the previous batch.
+ #[serde(default)]
+ search_after: Option<Vec<Value>>,
+ #[serde(default)]
+ error_count: usize,
+ #[serde(default)]
+ last_error: Option<String>,
+ #[serde(default)]
+ processing_stats: ProcessingStats,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct ProcessingStats {
+ #[serde(default)]
+ total_bytes_processed: u64,
+ /// Running cumulative average over the connector's lifetime, persisted and accumulated
+ /// across restarts. Reflects long-term throughput baseline, not session-only average.
+ #[serde(default)]
+ avg_batch_processing_time_ms: f64,
+ #[serde(default)]
+ last_successful_poll: Option<DateTime<Utc>>,
+ #[serde(default)]
+ empty_polls_count: usize,
+ #[serde(default)]
+ successful_polls_count: usize,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct StateConfig {
+ #[serde(default)]
+ pub enabled: bool,
+ pub storage_type: Option<String>,
+ pub storage_config: Option<Value>,
+ pub state_id: Option<String>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct OpenSearchSourceConfig {
+ pub url: String,
+ pub index: String,
+ pub username: Option<String>,
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_optional_secret")]
+ pub password: Option<SecretString>,
+ pub query: Option<Value>,
+ pub polling_interval: Option<String>,
+ pub batch_size: Option<usize>,
+ pub timestamp_field: Option<String>,
+ #[serde(default)]
+ pub verbose_logging: bool,
+ pub state: Option<StateConfig>,
+ /// Total poll-phase attempt count (not retry count); 1 = no retries, 3 = 2 retries.
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+ pub retry_max_delay: Option<String>,
+ /// Total open-phase attempt count (not retry count); 1 = no retries, 5 = 4 retries.
+ pub max_open_retries: Option<u32>,
+ pub open_retry_max_delay: Option<String>,
+ pub circuit_breaker_threshold: Option<u32>,
+ pub circuit_breaker_cool_down: Option<String>,
+}
+
+#[derive(Debug)]
+pub struct OpenSearchSource {
+ id: u32,
+ config: OpenSearchSourceConfig,
+ client: Option<OpenSearch>,
+ polling_interval: Duration,
+ /// Pre-built query body (query + size + sort). Set once in `open()` after config validation.
+ /// `None` before `open()` and after a failed `open()`.
+ search_body_base: Option<Value>,
+ verbose: bool,
+ max_retries: u32,
+ retry_delay: Duration,
+ retry_max_delay: Duration,
+ max_open_retries: u32,
+ open_retry_max_delay: Duration,
+ circuit_breaker: Arc<CircuitBreaker>,
+ state: Mutex<State>,
+ /// `Some(cause)` when runtime state restore was rejected; `None` means restore succeeded.
+ state_restore_error: Option<String>,
+ /// True when `new()` restored a valid runtime `ConnectorState`. File mirror must not override it.
+ runtime_state_restored: bool,
+}
+
+struct SearchOutcome {
+ messages: Vec<ProducedMessage>,
+ search_after: Option<Vec<Value>>,
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ batch_bytes: u64,
+}
+
+impl OpenSearchSource {
+ pub fn new(id: u32, config: OpenSearchSourceConfig, state: Option<ConnectorState>) -> Self {
+ let polling_interval =
+ parse_duration(config.polling_interval.as_deref(), DEFAULT_POLLING_INTERVAL);
+ let verbose = config.verbose_logging;
+ let (restored_state, state_restore_error, runtime_state_restored) =
+ restore_state(id, state);
+
+ let cb_threshold = config
+ .circuit_breaker_threshold
+ .unwrap_or(DEFAULT_CB_THRESHOLD);
+ let cb_cool_down = parse_duration(
+ config.circuit_breaker_cool_down.as_deref(),
+ DEFAULT_CB_COOL_DOWN,
+ );
+ let circuit_breaker = Arc::new(CircuitBreaker::new(cb_threshold, cb_cool_down));
+ let max_retries =
+ normalized_max_attempts(config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES));
+ let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+ let retry_max_delay =
+ parse_duration(config.retry_max_delay.as_deref(), DEFAULT_RETRY_MAX_DELAY);
+ let max_open_retries =
+ normalized_max_attempts(config.max_open_retries.unwrap_or(DEFAULT_MAX_OPEN_RETRIES));
+ let open_retry_max_delay = parse_duration(
+ config.open_retry_max_delay.as_deref(),
+ DEFAULT_OPEN_RETRY_MAX_DELAY,
+ );
+
+ OpenSearchSource {
+ id,
+ config,
+ max_retries,
+ retry_delay,
+ retry_max_delay,
+ max_open_retries,
+ open_retry_max_delay,
+ circuit_breaker,
+ client: None,
+ polling_interval,
+ search_body_base: None,
+ verbose,
+ state: Mutex::new(restored_state),
+ state_restore_error,
+ runtime_state_restored,
+ }
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn batch_size(&self) -> usize {
+ self.config.batch_size.unwrap_or(DEFAULT_BATCH_SIZE)
+ }
+
+ fn timestamp_field(&self) -> &str {
+ self.config
+ .timestamp_field
+ .as_deref()
+ .expect("timestamp_field validated at open()")
Review Comment:
`expect` on a prod path. it's unreachable today (`validate_open_config` rejects `None`/empty before any caller), but nothing type-enforces that. `poll()` runs in a spawned task inside the plugin, so a panic here wouldn't crash anything visibly - it kills the polling task and the connector silently stops. an `Err` gets logged and the loop continues. resolving the field once after validation (or returning `Err`) avoids that failure mode if a future refactor reorders things.
##########
core/connectors/sources/opensearch_source/src/lib.rs:
##########
@@ -0,0 +1,1136 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::retry::{CircuitBreaker, parse_duration};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use opensearch::{
+ OpenSearch, SearchParts,
+ auth::Credentials,
+ http::{Url, transport::TransportBuilder},
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use serde_json::{Value, json};
+use std::sync::Arc;
+use std::time::Duration;
+use tokio::{sync::Mutex, time::sleep};
+use tracing::{debug, error, info, warn};
+
+mod retry;
+mod state_manager;
+use crate::retry::{
+ DEFAULT_CB_COOL_DOWN, DEFAULT_CB_THRESHOLD, DEFAULT_MAX_OPEN_RETRIES, DEFAULT_MAX_RETRIES,
+ DEFAULT_OPEN_RETRY_MAX_DELAY, DEFAULT_RETRY_DELAY, DEFAULT_RETRY_MAX_DELAY, RetryBackoff,
+ is_transient_status, normalized_max_attempts, sleep_before_retry,
+};
+use crate::state_manager::{SOURCE_STATE_VERSION, SourceState, validate_state_storage_config};
+
+source_connector!(OpenSearchSource);
+
+const CONNECTOR_NAME: &str = "OpenSearch source";
+const DEFAULT_POLLING_INTERVAL: &str = "10s";
+const DEFAULT_BATCH_SIZE: usize = 100;
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct State {
+ #[serde(default)]
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ #[serde(default, alias = "total_documents_fetched")]
+ total_documents_published: usize,
+ #[serde(default)]
+ poll_count: usize,
+ /// OpenSearch `search_after` tuple from the last hit in the previous batch.
+ #[serde(default)]
+ search_after: Option<Vec<Value>>,
+ #[serde(default)]
+ error_count: usize,
+ #[serde(default)]
+ last_error: Option<String>,
+ #[serde(default)]
+ processing_stats: ProcessingStats,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct ProcessingStats {
+ #[serde(default)]
+ total_bytes_processed: u64,
+ /// Running cumulative average over the connector's lifetime, persisted and accumulated
+ /// across restarts. Reflects long-term throughput baseline, not session-only average.
+ #[serde(default)]
+ avg_batch_processing_time_ms: f64,
+ #[serde(default)]
+ last_successful_poll: Option<DateTime<Utc>>,
+ #[serde(default)]
+ empty_polls_count: usize,
+ #[serde(default)]
+ successful_polls_count: usize,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct StateConfig {
+ #[serde(default)]
+ pub enabled: bool,
+ pub storage_type: Option<String>,
+ pub storage_config: Option<Value>,
+ pub state_id: Option<String>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct OpenSearchSourceConfig {
+ pub url: String,
+ pub index: String,
+ pub username: Option<String>,
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_optional_secret")]
+ pub password: Option<SecretString>,
+ pub query: Option<Value>,
+ pub polling_interval: Option<String>,
+ pub batch_size: Option<usize>,
+ pub timestamp_field: Option<String>,
+ #[serde(default)]
+ pub verbose_logging: bool,
+ pub state: Option<StateConfig>,
+ /// Total poll-phase attempt count (not retry count); 1 = no retries, 3 = 2 retries.
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+ pub retry_max_delay: Option<String>,
+ /// Total open-phase attempt count (not retry count); 1 = no retries, 5 = 4 retries.
+ pub max_open_retries: Option<u32>,
+ pub open_retry_max_delay: Option<String>,
+ pub circuit_breaker_threshold: Option<u32>,
+ pub circuit_breaker_cool_down: Option<String>,
+}
+
+#[derive(Debug)]
+pub struct OpenSearchSource {
+ id: u32,
+ config: OpenSearchSourceConfig,
+ client: Option<OpenSearch>,
+ polling_interval: Duration,
+ /// Pre-built query body (query + size + sort). Set once in `open()` after config validation.
+ /// `None` before `open()` and after a failed `open()`.
+ search_body_base: Option<Value>,
+ verbose: bool,
+ max_retries: u32,
+ retry_delay: Duration,
+ retry_max_delay: Duration,
+ max_open_retries: u32,
+ open_retry_max_delay: Duration,
+ circuit_breaker: Arc<CircuitBreaker>,
+ state: Mutex<State>,
+ /// `Some(cause)` when runtime state restore was rejected; `None` means restore succeeded.
+ state_restore_error: Option<String>,
+ /// True when `new()` restored a valid runtime `ConnectorState`. File mirror must not override it.
+ runtime_state_restored: bool,
+}
+
+struct SearchOutcome {
+ messages: Vec<ProducedMessage>,
+ search_after: Option<Vec<Value>>,
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ batch_bytes: u64,
+}
+
+impl OpenSearchSource {
+ pub fn new(id: u32, config: OpenSearchSourceConfig, state: Option<ConnectorState>) -> Self {
+ let polling_interval =
+ parse_duration(config.polling_interval.as_deref(), DEFAULT_POLLING_INTERVAL);
+ let verbose = config.verbose_logging;
+ let (restored_state, state_restore_error, runtime_state_restored) =
+ restore_state(id, state);
+
+ let cb_threshold = config
+ .circuit_breaker_threshold
+ .unwrap_or(DEFAULT_CB_THRESHOLD);
+ let cb_cool_down = parse_duration(
+ config.circuit_breaker_cool_down.as_deref(),
+ DEFAULT_CB_COOL_DOWN,
+ );
+ let circuit_breaker = Arc::new(CircuitBreaker::new(cb_threshold, cb_cool_down));
+ let max_retries =
+ normalized_max_attempts(config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES));
+ let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+ let retry_max_delay =
+ parse_duration(config.retry_max_delay.as_deref(), DEFAULT_RETRY_MAX_DELAY);
+ let max_open_retries =
+ normalized_max_attempts(config.max_open_retries.unwrap_or(DEFAULT_MAX_OPEN_RETRIES));
+ let open_retry_max_delay = parse_duration(
+ config.open_retry_max_delay.as_deref(),
+ DEFAULT_OPEN_RETRY_MAX_DELAY,
+ );
+
+ OpenSearchSource {
+ id,
+ config,
+ max_retries,
+ retry_delay,
+ retry_max_delay,
+ max_open_retries,
+ open_retry_max_delay,
+ circuit_breaker,
+ client: None,
+ polling_interval,
+ search_body_base: None,
+ verbose,
+ state: Mutex::new(restored_state),
+ state_restore_error,
+ runtime_state_restored,
+ }
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn batch_size(&self) -> usize {
+ self.config.batch_size.unwrap_or(DEFAULT_BATCH_SIZE)
+ }
+
+ fn timestamp_field(&self) -> &str {
+ self.config
+ .timestamp_field
+ .as_deref()
+ .expect("timestamp_field validated at open()")
+ }
+
+ fn get_state_id(&self) -> String {
+ self.config
+ .state
+ .as_ref()
+ .and_then(|s| s.state_id.clone())
+ .unwrap_or_else(|| format!("opensearch_source_{}", self.id))
+ }
+
+ pub(crate) async fn internal_state_to_source_state(&self) -> Result<SourceState, Error> {
+ let state = self.state.lock().await;
+ let data = serde_json::to_value(&*state).map_err(|error| {
+ Error::Serialization(format!("Failed to serialize connector state: {error}"))
+ })?;
+
+ Ok(SourceState {
+ id: self.get_state_id(),
+ last_updated: Utc::now(),
+ version: SOURCE_STATE_VERSION,
+ data,
+ metadata: Some(json!({
+ "connector_type": "opensearch_source",
+ "connector_id": self.id,
+ "index": self.config.index,
+ "url": self.config.url,
Review Comment:
`config.url` is written verbatim into file-state metadata and lands on disk as plaintext json when file state is enabled. a url with embedded userinfo (`http://user:pass@host`) puts the credential at rest. the metadata block is informational and unused on restore (`source_state_to_internal_state` reads only `data` + `version`), so dropping `url` (or stripping userinfo) is free. `elasticsearch_source` has the same pattern, no need to copy it forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "kriti-sc (via GitHub)" <gi...@apache.org>.
kriti-sc commented on code in PR #3515:
URL: https://github.com/apache/iggy/pull/3515#discussion_r3521229978
##########
core/connectors/sources/opensearch_source/src/lib.rs:
##########
@@ -0,0 +1,1135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::retry::{CircuitBreaker, parse_duration};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use opensearch::{
+ OpenSearch, SearchParts,
+ auth::Credentials,
+ http::{Url, transport::TransportBuilder},
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use serde_json::{Value, json};
+use std::sync::Arc;
+use std::time::Duration;
+use tokio::{sync::Mutex, time::sleep};
+use tracing::{debug, error, info, warn};
+
+mod retry;
+mod state_manager;
+use crate::retry::{
+ DEFAULT_CB_COOL_DOWN, DEFAULT_CB_THRESHOLD, DEFAULT_MAX_OPEN_RETRIES, DEFAULT_MAX_RETRIES,
+ DEFAULT_OPEN_RETRY_MAX_DELAY, DEFAULT_RETRY_DELAY, DEFAULT_RETRY_MAX_DELAY, RetryBackoff,
+ is_transient_status, normalized_max_attempts, sleep_before_retry,
+};
+use crate::state_manager::{SOURCE_STATE_VERSION, SourceState, validate_state_storage_config};
+
+source_connector!(OpenSearchSource);
+
+const CONNECTOR_NAME: &str = "OpenSearch source";
+const DEFAULT_POLLING_INTERVAL: &str = "10s";
+const DEFAULT_BATCH_SIZE: usize = 100;
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct State {
+ #[serde(default)]
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ #[serde(default, alias = "total_documents_fetched")]
+ total_documents_published: usize,
+ #[serde(default)]
+ poll_count: usize,
+ /// OpenSearch `search_after` tuple from the last hit in the previous batch.
+ #[serde(default)]
+ search_after: Option<Vec<Value>>,
+ #[serde(default)]
+ error_count: usize,
+ #[serde(default)]
+ last_error: Option<String>,
+ #[serde(default)]
+ processing_stats: ProcessingStats,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct ProcessingStats {
+ #[serde(default)]
+ total_bytes_processed: u64,
+ /// Running cumulative average over the connector's lifetime, persisted and accumulated
+ /// across restarts. Reflects long-term throughput baseline, not session-only average.
+ #[serde(default)]
+ avg_batch_processing_time_ms: f64,
+ #[serde(default)]
+ last_successful_poll: Option<DateTime<Utc>>,
+ #[serde(default)]
+ empty_polls_count: usize,
+ #[serde(default)]
+ successful_polls_count: usize,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct StateConfig {
+ #[serde(default)]
+ pub enabled: bool,
+ pub storage_type: Option<String>,
+ pub storage_config: Option<Value>,
+ pub state_id: Option<String>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct OpenSearchSourceConfig {
+ pub url: String,
+ pub index: String,
+ pub username: Option<String>,
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_optional_secret")]
+ pub password: Option<SecretString>,
+ pub query: Option<Value>,
+ pub polling_interval: Option<String>,
+ pub batch_size: Option<usize>,
+ pub timestamp_field: Option<String>,
+ #[serde(default)]
+ pub verbose_logging: bool,
+ pub state: Option<StateConfig>,
+ /// Total poll-phase attempt count (not retry count); 1 = no retries, 3 = 2 retries.
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+ pub retry_max_delay: Option<String>,
+ /// Total open-phase attempt count (not retry count); 1 = no retries, 5 = 4 retries.
+ pub max_open_retries: Option<u32>,
+ pub open_retry_max_delay: Option<String>,
+ pub circuit_breaker_threshold: Option<u32>,
+ pub circuit_breaker_cool_down: Option<String>,
+}
+
+#[derive(Debug)]
+pub struct OpenSearchSource {
+ id: u32,
+ config: OpenSearchSourceConfig,
+ client: Option<OpenSearch>,
+ polling_interval: Duration,
+ /// Pre-built query body (query + size + sort). Set once in `open()` after config validation.
+ /// `None` before `open()` and after a failed `open()`.
+ search_body_base: Option<Value>,
+ verbose: bool,
+ max_retries: u32,
+ retry_delay: Duration,
+ retry_max_delay: Duration,
+ max_open_retries: u32,
+ open_retry_max_delay: Duration,
+ circuit_breaker: Arc<CircuitBreaker>,
+ state: Mutex<State>,
+ /// `Some(cause)` when runtime state restore was rejected; `None` means restore succeeded.
+ state_restore_error: Option<String>,
+ /// True when `new()` restored a valid runtime `ConnectorState`. File mirror must not override it.
+ runtime_state_restored: bool,
+}
+
+struct SearchOutcome {
+ messages: Vec<ProducedMessage>,
+ search_after: Option<Vec<Value>>,
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ batch_bytes: u64,
+}
+
+impl OpenSearchSource {
+ pub fn new(id: u32, config: OpenSearchSourceConfig, state: Option<ConnectorState>) -> Self {
+ let polling_interval =
+ parse_duration(config.polling_interval.as_deref(), DEFAULT_POLLING_INTERVAL);
+ let verbose = config.verbose_logging;
+ let (restored_state, state_restore_error, runtime_state_restored) =
+ restore_state(id, state);
+
+ let cb_threshold = config
+ .circuit_breaker_threshold
+ .unwrap_or(DEFAULT_CB_THRESHOLD);
+ let cb_cool_down = parse_duration(
+ config.circuit_breaker_cool_down.as_deref(),
+ DEFAULT_CB_COOL_DOWN,
+ );
+ let circuit_breaker = Arc::new(CircuitBreaker::new(cb_threshold, cb_cool_down));
+ let max_retries =
+ normalized_max_attempts(config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES));
+ let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+ let retry_max_delay =
+ parse_duration(config.retry_max_delay.as_deref(), DEFAULT_RETRY_MAX_DELAY);
+ let max_open_retries =
+ normalized_max_attempts(config.max_open_retries.unwrap_or(DEFAULT_MAX_OPEN_RETRIES));
+ let open_retry_max_delay = parse_duration(
+ config.open_retry_max_delay.as_deref(),
+ DEFAULT_OPEN_RETRY_MAX_DELAY,
+ );
+
+ OpenSearchSource {
+ id,
+ config,
+ max_retries,
+ retry_delay,
+ retry_max_delay,
+ max_open_retries,
+ open_retry_max_delay,
+ circuit_breaker,
+ client: None,
+ polling_interval,
+ search_body_base: None,
+ verbose,
+ state: Mutex::new(restored_state),
+ state_restore_error,
+ runtime_state_restored,
+ }
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn batch_size(&self) -> usize {
+ self.config.batch_size.unwrap_or(DEFAULT_BATCH_SIZE)
+ }
+
+ fn timestamp_field(&self) -> Result<&str, Error> {
+ self.config
+ .timestamp_field
+ .as_deref()
+ .filter(|s| !s.is_empty())
+ .ok_or_else(|| Error::InvalidConfigValue("timestamp_field is required".to_string()))
+ }
+
+ fn get_state_id(&self) -> String {
+ self.config
+ .state
+ .as_ref()
+ .and_then(|s| s.state_id.clone())
+ .unwrap_or_else(|| format!("opensearch_source_{}", self.id))
+ }
+
+ pub(crate) async fn internal_state_to_source_state(&self) -> Result<SourceState, Error> {
+ let state = self.state.lock().await;
+ let data = serde_json::to_value(&*state).map_err(|error| {
+ Error::Serialization(format!("Failed to serialize connector state: {error}"))
+ })?;
+
+ Ok(SourceState {
+ id: self.get_state_id(),
+ last_updated: Utc::now(),
+ version: SOURCE_STATE_VERSION,
+ data,
+ metadata: Some(json!({
+ "connector_type": "opensearch_source",
+ "connector_id": self.id,
+ "index": self.config.index,
+ })),
+ })
+ }
+
+ pub(crate) async fn source_state_to_internal_state(
+ &mut self,
+ source_state: SourceState,
+ ) -> Result<(), Error> {
+ if source_state.version != SOURCE_STATE_VERSION {
+ return Err(Error::Serialization(format!(
+ "unsupported file state version {}, expected {SOURCE_STATE_VERSION}",
+ source_state.version
+ )));
+ }
+
+ let restored: State = serde_json::from_value(source_state.data).map_err(|error| {
+ Error::Serialization(format!("Failed to deserialize connector state: {error}"))
+ })?;
+
+ let mut state = self.state.lock().await;
+ *state = restored;
+ Ok(())
+ }
+
+ async fn create_client(&self) -> Result<OpenSearch, Error> {
+ let url = Url::parse(&self.config.url).map_err(|error| {
+ Error::InvalidConfigValue(format!("Invalid OpenSearch URL: {error}"))
+ })?;
+
+ let conn_pool = opensearch::http::transport::SingleNodeConnectionPool::new(url);
+ let mut transport_builder = TransportBuilder::new(conn_pool);
+
+ if let (Some(username), Some(password)) = (&self.config.username, &self.config.password) {
+ let credentials =
+ Credentials::Basic(username.clone(), password.expose_secret().to_string());
+ transport_builder = transport_builder.auth(credentials);
+ }
+
+ let transport = transport_builder
+ .build()
+ .map_err(|error| Error::InitError(format!("Failed to build transport: {error}")))?;
+
+ Ok(OpenSearch::new(transport))
+ }
+
+ async fn check_index_exists_with_retry(&self, client: &OpenSearch) -> Result<(), Error> {
+ let max_attempts = self.max_open_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let response = match client
+ .indices()
+ .exists(opensearch::indices::IndicesExistsParts::Index(&[&self
+ .config
+ .index]))
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ // retry_delay is shared with the poll phase (no separate open-phase
+ // base delay); open_retry_max_delay caps the growth independently.
+ // Same pattern as the InfluxDB source connector.
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::InitError(format!(
+ "Failed to check index existence: {error}"
+ )));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(());
+ }
+
+ let status = response.status_code().as_u16();
+ if status == 404 {
+ return Err(Error::InitError(format!(
+ "Index '{}' does not exist or is not accessible",
+ self.config.index
+ )));
+ }
+
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::InitError(format!(
+ "Index '{}' check failed: HTTP {status}: {error_text}",
+ self.config.index
+ )));
+ }
+ }
+
+ async fn send_search_with_retry(
+ &self,
+ client: &OpenSearch,
+ mut search_body: Value,
+ ) -> Result<opensearch::http::response::Response, Error> {
+ let max_attempts = self.max_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let body = if attempt < max_attempts {
+ search_body.clone()
+ } else {
+ std::mem::take(&mut search_body)
+ };
+ let response = match client
+ .search(SearchParts::Index(&[&self.config.index]))
+ .body(body)
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::Storage(format!("Failed to execute search: {error}")));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(response);
+ }
+
+ let status = response.status_code().as_u16();
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::Storage(format!(
+ "Search request failed: {error_text}"
+ )));
+ }
+ }
+
+ async fn search_documents(&self, client: &OpenSearch) -> Result<SearchOutcome, Error> {
+ let state = self.state.lock().await;
+ let search_after = state.search_after.clone();
+ drop(state);
+
+ let timestamp_field = self.timestamp_field()?;
+
+ let mut search_body = self
+ .search_body_base
+ .as_ref()
+ .ok_or_else(|| {
+ Error::Connection("connector not initialized; call open() first".to_string())
+ })?
+ .clone();
+
+ if let Some(cursor) = search_after {
+ search_body["search_after"] = json!(cursor);
+ }
+
+ let response = self.send_search_with_retry(client, search_body).await?;
+
+ let mut response_body: Value = response
+ .json()
+ .await
+ .map_err(|e| Error::Storage(format!("Failed to parse search response: {e}")))?;
+
+ let hits: Vec<Value> = response_body
+ .get_mut("hits")
+ .and_then(|h| h.get_mut("hits"))
+ .and_then(|arr| arr.as_array_mut())
+ .map(std::mem::take)
+ .unwrap_or_default();
+
+ let mut messages = Vec::with_capacity(hits.len());
+ let mut batch_bytes = 0u64;
+ let mut last_sort = None;
+ let mut last_poll_timestamp = None;
+
+ for hit in &hits {
+ let Some(sort) = hit
+ .get("sort")
+ .and_then(|s| s.as_array())
+ .filter(|a| !a.is_empty())
+ else {
+ warn!(
+ connector_id = self.id,
+ hit_id = hit.get("_id").and_then(|value| value.as_str()),
+ "Skipping OpenSearch hit without sort tuple; document will not be published"
+ );
+ continue;
+ };
+
+ last_sort = Some(sort);
Review Comment:
correct me if i am wrong, but it appears that cursor advances before confirming if a doc is published. ideally, the cursor should advance only after the doc is published. this is an ordering issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #3515:
URL: https://github.com/apache/iggy/pull/3515#discussion_r3521553423
##########
core/connectors/sources/opensearch_source/src/lib.rs:
##########
@@ -0,0 +1,1135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use async_trait::async_trait;
+use iggy_common::{DateTime, Utc};
+use iggy_connector_sdk::retry::{CircuitBreaker, parse_duration};
+use iggy_connector_sdk::{
+ ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, source_connector,
+};
+use opensearch::{
+ OpenSearch, SearchParts,
+ auth::Credentials,
+ http::{Url, transport::TransportBuilder},
+};
+use secrecy::{ExposeSecret, SecretString};
+use serde::{Deserialize, Serialize};
+use serde_json::{Value, json};
+use std::sync::Arc;
+use std::time::Duration;
+use tokio::{sync::Mutex, time::sleep};
+use tracing::{debug, error, info, warn};
+
+mod retry;
+mod state_manager;
+use crate::retry::{
+ DEFAULT_CB_COOL_DOWN, DEFAULT_CB_THRESHOLD, DEFAULT_MAX_OPEN_RETRIES, DEFAULT_MAX_RETRIES,
+ DEFAULT_OPEN_RETRY_MAX_DELAY, DEFAULT_RETRY_DELAY, DEFAULT_RETRY_MAX_DELAY, RetryBackoff,
+ is_transient_status, normalized_max_attempts, sleep_before_retry,
+};
+use crate::state_manager::{SOURCE_STATE_VERSION, SourceState, validate_state_storage_config};
+
+source_connector!(OpenSearchSource);
+
+const CONNECTOR_NAME: &str = "OpenSearch source";
+const DEFAULT_POLLING_INTERVAL: &str = "10s";
+const DEFAULT_BATCH_SIZE: usize = 100;
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct State {
+ #[serde(default)]
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ #[serde(default, alias = "total_documents_fetched")]
+ total_documents_published: usize,
+ #[serde(default)]
+ poll_count: usize,
+ /// OpenSearch `search_after` tuple from the last hit in the previous batch.
+ #[serde(default)]
+ search_after: Option<Vec<Value>>,
+ #[serde(default)]
+ error_count: usize,
+ #[serde(default)]
+ last_error: Option<String>,
+ #[serde(default)]
+ processing_stats: ProcessingStats,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+struct ProcessingStats {
+ #[serde(default)]
+ total_bytes_processed: u64,
+ /// Running cumulative average over the connector's lifetime, persisted and accumulated
+ /// across restarts. Reflects long-term throughput baseline, not session-only average.
+ #[serde(default)]
+ avg_batch_processing_time_ms: f64,
+ #[serde(default)]
+ last_successful_poll: Option<DateTime<Utc>>,
+ #[serde(default)]
+ empty_polls_count: usize,
+ #[serde(default)]
+ successful_polls_count: usize,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct StateConfig {
+ #[serde(default)]
+ pub enabled: bool,
+ pub storage_type: Option<String>,
+ pub storage_config: Option<Value>,
+ pub state_id: Option<String>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct OpenSearchSourceConfig {
+ pub url: String,
+ pub index: String,
+ pub username: Option<String>,
+ #[serde(serialize_with = "iggy_common::serde_secret::serialize_optional_secret")]
+ pub password: Option<SecretString>,
+ pub query: Option<Value>,
+ pub polling_interval: Option<String>,
+ pub batch_size: Option<usize>,
+ pub timestamp_field: Option<String>,
+ #[serde(default)]
+ pub verbose_logging: bool,
+ pub state: Option<StateConfig>,
+ /// Total poll-phase attempt count (not retry count); 1 = no retries, 3 = 2 retries.
+ pub max_retries: Option<u32>,
+ pub retry_delay: Option<String>,
+ pub retry_max_delay: Option<String>,
+ /// Total open-phase attempt count (not retry count); 1 = no retries, 5 = 4 retries.
+ pub max_open_retries: Option<u32>,
+ pub open_retry_max_delay: Option<String>,
+ pub circuit_breaker_threshold: Option<u32>,
+ pub circuit_breaker_cool_down: Option<String>,
+}
+
+#[derive(Debug)]
+pub struct OpenSearchSource {
+ id: u32,
+ config: OpenSearchSourceConfig,
+ client: Option<OpenSearch>,
+ polling_interval: Duration,
+ /// Pre-built query body (query + size + sort). Set once in `open()` after config validation.
+ /// `None` before `open()` and after a failed `open()`.
+ search_body_base: Option<Value>,
+ verbose: bool,
+ max_retries: u32,
+ retry_delay: Duration,
+ retry_max_delay: Duration,
+ max_open_retries: u32,
+ open_retry_max_delay: Duration,
+ circuit_breaker: Arc<CircuitBreaker>,
+ state: Mutex<State>,
+ /// `Some(cause)` when runtime state restore was rejected; `None` means restore succeeded.
+ state_restore_error: Option<String>,
+ /// True when `new()` restored a valid runtime `ConnectorState`. File mirror must not override it.
+ runtime_state_restored: bool,
+}
+
+struct SearchOutcome {
+ messages: Vec<ProducedMessage>,
+ search_after: Option<Vec<Value>>,
+ last_poll_timestamp: Option<DateTime<Utc>>,
+ batch_bytes: u64,
+}
+
+impl OpenSearchSource {
+ pub fn new(id: u32, config: OpenSearchSourceConfig, state: Option<ConnectorState>) -> Self {
+ let polling_interval =
+ parse_duration(config.polling_interval.as_deref(), DEFAULT_POLLING_INTERVAL);
+ let verbose = config.verbose_logging;
+ let (restored_state, state_restore_error, runtime_state_restored) =
+ restore_state(id, state);
+
+ let cb_threshold = config
+ .circuit_breaker_threshold
+ .unwrap_or(DEFAULT_CB_THRESHOLD);
+ let cb_cool_down = parse_duration(
+ config.circuit_breaker_cool_down.as_deref(),
+ DEFAULT_CB_COOL_DOWN,
+ );
+ let circuit_breaker = Arc::new(CircuitBreaker::new(cb_threshold, cb_cool_down));
+ let max_retries =
+ normalized_max_attempts(config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES));
+ let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+ let retry_max_delay =
+ parse_duration(config.retry_max_delay.as_deref(), DEFAULT_RETRY_MAX_DELAY);
+ let max_open_retries =
+ normalized_max_attempts(config.max_open_retries.unwrap_or(DEFAULT_MAX_OPEN_RETRIES));
+ let open_retry_max_delay = parse_duration(
+ config.open_retry_max_delay.as_deref(),
+ DEFAULT_OPEN_RETRY_MAX_DELAY,
+ );
+
+ OpenSearchSource {
+ id,
+ config,
+ max_retries,
+ retry_delay,
+ retry_max_delay,
+ max_open_retries,
+ open_retry_max_delay,
+ circuit_breaker,
+ client: None,
+ polling_interval,
+ search_body_base: None,
+ verbose,
+ state: Mutex::new(restored_state),
+ state_restore_error,
+ runtime_state_restored,
+ }
+ }
+
+ fn serialize_state(&self, state: &State) -> Option<ConnectorState> {
+ ConnectorState::serialize(state, CONNECTOR_NAME, self.id)
+ }
+
+ fn batch_size(&self) -> usize {
+ self.config.batch_size.unwrap_or(DEFAULT_BATCH_SIZE)
+ }
+
+ fn timestamp_field(&self) -> Result<&str, Error> {
+ self.config
+ .timestamp_field
+ .as_deref()
+ .filter(|s| !s.is_empty())
+ .ok_or_else(|| Error::InvalidConfigValue("timestamp_field is required".to_string()))
+ }
+
+ fn get_state_id(&self) -> String {
+ self.config
+ .state
+ .as_ref()
+ .and_then(|s| s.state_id.clone())
+ .unwrap_or_else(|| format!("opensearch_source_{}", self.id))
+ }
+
+ pub(crate) async fn internal_state_to_source_state(&self) -> Result<SourceState, Error> {
+ let state = self.state.lock().await;
+ let data = serde_json::to_value(&*state).map_err(|error| {
+ Error::Serialization(format!("Failed to serialize connector state: {error}"))
+ })?;
+
+ Ok(SourceState {
+ id: self.get_state_id(),
+ last_updated: Utc::now(),
+ version: SOURCE_STATE_VERSION,
+ data,
+ metadata: Some(json!({
+ "connector_type": "opensearch_source",
+ "connector_id": self.id,
+ "index": self.config.index,
+ })),
+ })
+ }
+
+ pub(crate) async fn source_state_to_internal_state(
+ &mut self,
+ source_state: SourceState,
+ ) -> Result<(), Error> {
+ if source_state.version != SOURCE_STATE_VERSION {
+ return Err(Error::Serialization(format!(
+ "unsupported file state version {}, expected {SOURCE_STATE_VERSION}",
+ source_state.version
+ )));
+ }
+
+ let restored: State = serde_json::from_value(source_state.data).map_err(|error| {
+ Error::Serialization(format!("Failed to deserialize connector state: {error}"))
+ })?;
+
+ let mut state = self.state.lock().await;
+ *state = restored;
+ Ok(())
+ }
+
+ async fn create_client(&self) -> Result<OpenSearch, Error> {
+ let url = Url::parse(&self.config.url).map_err(|error| {
+ Error::InvalidConfigValue(format!("Invalid OpenSearch URL: {error}"))
+ })?;
+
+ let conn_pool = opensearch::http::transport::SingleNodeConnectionPool::new(url);
+ let mut transport_builder = TransportBuilder::new(conn_pool);
+
+ if let (Some(username), Some(password)) = (&self.config.username, &self.config.password) {
+ let credentials =
+ Credentials::Basic(username.clone(), password.expose_secret().to_string());
+ transport_builder = transport_builder.auth(credentials);
+ }
+
+ let transport = transport_builder
+ .build()
+ .map_err(|error| Error::InitError(format!("Failed to build transport: {error}")))?;
+
+ Ok(OpenSearch::new(transport))
+ }
+
+ async fn check_index_exists_with_retry(&self, client: &OpenSearch) -> Result<(), Error> {
+ let max_attempts = self.max_open_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let response = match client
+ .indices()
+ .exists(opensearch::indices::IndicesExistsParts::Index(&[&self
+ .config
+ .index]))
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ // retry_delay is shared with the poll phase (no separate open-phase
+ // base delay); open_retry_max_delay caps the growth independently.
+ // Same pattern as the InfluxDB source connector.
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::InitError(format!(
+ "Failed to check index existence: {error}"
+ )));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(());
+ }
+
+ let status = response.status_code().as_u16();
+ if status == 404 {
+ return Err(Error::InitError(format!(
+ "Index '{}' does not exist or is not accessible",
+ self.config.index
+ )));
+ }
+
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "index_exists",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.open_retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::InitError(format!(
+ "Index '{}' check failed: HTTP {status}: {error_text}",
+ self.config.index
+ )));
+ }
+ }
+
+ async fn send_search_with_retry(
+ &self,
+ client: &OpenSearch,
+ mut search_body: Value,
+ ) -> Result<opensearch::http::response::Response, Error> {
+ let max_attempts = self.max_retries;
+ let mut attempt = 0u32;
+
+ loop {
+ attempt += 1;
+ let body = if attempt < max_attempts {
+ search_body.clone()
+ } else {
+ std::mem::take(&mut search_body)
+ };
+ let response = match client
+ .search(SearchParts::Index(&[&self.config.index]))
+ .body(body)
+ .send()
+ .await
+ {
+ Ok(response) => response,
+ Err(error) => {
+ if attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ None,
+ &error.to_string(),
+ )
+ .await;
+ continue;
+ }
+ return Err(Error::Storage(format!("Failed to execute search: {error}")));
+ }
+ };
+
+ if response.status_code().is_success() {
+ return Ok(response);
+ }
+
+ let status = response.status_code().as_u16();
+ let retry_after = response
+ .headers()
+ .get("retry-after")
+ .and_then(|value| value.to_str().ok())
+ .map(str::to_owned);
+ let error_text = response
+ .text()
+ .await
+ .unwrap_or_else(|_| "unknown error".to_string());
+
+ if is_transient_status(status) && attempt < max_attempts {
+ sleep_before_retry(
+ "search",
+ self.id,
+ attempt,
+ max_attempts,
+ &RetryBackoff {
+ delay: self.retry_delay,
+ max_delay: self.retry_max_delay,
+ },
+ retry_after.as_deref(),
+ &format!("HTTP {status}: {error_text}"),
+ )
+ .await;
+ continue;
+ }
+
+ return Err(Error::Storage(format!(
+ "Search request failed: {error_text}"
+ )));
+ }
+ }
+
+ async fn search_documents(&self, client: &OpenSearch) -> Result<SearchOutcome, Error> {
+ let state = self.state.lock().await;
+ let search_after = state.search_after.clone();
+ drop(state);
+
+ let timestamp_field = self.timestamp_field()?;
+
+ let mut search_body = self
+ .search_body_base
+ .as_ref()
+ .ok_or_else(|| {
+ Error::Connection("connector not initialized; call open() first".to_string())
+ })?
+ .clone();
+
+ if let Some(cursor) = search_after {
+ search_body["search_after"] = json!(cursor);
+ }
+
+ let response = self.send_search_with_retry(client, search_body).await?;
+
+ let mut response_body: Value = response
+ .json()
+ .await
+ .map_err(|e| Error::Storage(format!("Failed to parse search response: {e}")))?;
+
+ let hits: Vec<Value> = response_body
+ .get_mut("hits")
+ .and_then(|h| h.get_mut("hits"))
+ .and_then(|arr| arr.as_array_mut())
+ .map(std::mem::take)
+ .unwrap_or_default();
+
+ let mut messages = Vec::with_capacity(hits.len());
+ let mut batch_bytes = 0u64;
+ let mut last_sort = None;
+ let mut last_poll_timestamp = None;
+
+ for hit in &hits {
+ let Some(sort) = hit
+ .get("sort")
+ .and_then(|s| s.as_array())
+ .filter(|a| !a.is_empty())
+ else {
+ warn!(
+ connector_id = self.id,
+ hit_id = hit.get("_id").and_then(|value| value.as_str()),
+ "Skipping OpenSearch hit without sort tuple; document will not be published"
+ );
+ continue;
+ };
+
+ last_sort = Some(sort);
+
+ let Some(source) = hit.get("_source") else {
+ warn!(
+ connector_id = self.id,
+ hit_id = hit.get("_id").and_then(|v| v.as_str()),
+ "Skipping OpenSearch hit without _source; document will not be published"
+ );
+ continue;
+ };
Review Comment:
Source could be empty, being an empty content, there is no value in sending for downstream applcations. Erroring out to flag as possible data missing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on PR #3515:
URL: https://github.com/apache/iggy/pull/3515#issuecomment-4884334257
/author
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): opensearch source connector [iggy]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3515:
URL: https://github.com/apache/iggy/pull/3515#issuecomment-5403233910
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.
If you need a review, please ensure CI is green and the PR is rebased on the latest master. Don't hesitate to ping the maintainers - either `@core` on Discord or by mentioning them directly here on the PR.
Thank you for your contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org