You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by pi...@apache.org on 2025/06/11 11:42:26 UTC

(iggy) branch connectors_api updated (4e63273a -> 23828750)

This is an automated email from the ASF dual-hosted git repository.

piotr pushed a change to branch connectors_api
in repository https://gitbox.apache.org/repos/asf/iggy.git


 discard 4e63273a add initial version of connectors http api
     add 0cfe495d feat(sdk): add background sending support to IggyProducer (#1838)
     new 23828750 add initial version of connectors http api

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (4e63273a)
            \
             N -- N -- N   refs/heads/connectors_api (23828750)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 Cargo.lock                                         |   2 +
 README.md                                          |   6 +-
 .../src/actors/consumer/high_level_backend.rs      |   4 +-
 core/bench/src/utils/batch_generator.rs            |   8 +
 core/common/src/error/iggy_error.rs                |  21 +-
 core/common/src/types/message/iggy_message.rs      |   2 +-
 core/connectors/runtime/src/configs.rs             |   4 +-
 core/connectors/runtime/src/sink.rs                |   2 +-
 core/connectors/runtime/src/source.rs              |  20 +-
 core/examples/src/multi-tenant/consumer/main.rs    |   4 +-
 core/examples/src/multi-tenant/producer/main.rs    |  22 +-
 core/examples/src/new-sdk/consumer/main.rs         |   2 +-
 core/examples/src/new-sdk/producer/main.rs         |  11 +-
 core/examples/src/sink-data-producer/main.rs       |  11 +-
 .../stream-builder/stream-consumer-config/main.rs  |   2 +-
 .../stream-builder/stream-producer-config/main.rs  |   4 +-
 core/integration/tests/mod.rs                      |   1 +
 .../src/lib.rs => integration/tests/sdk/mod.rs}    |   2 +-
 core/integration/tests/sdk/producer/background.rs  | 407 ++++++++++++++++
 core/integration/tests/sdk/producer/mod.rs         |  64 +++
 .../tests/server/scenarios/system_scenario.rs      |  10 +-
 core/integration/tests/streaming/get_by_offset.rs  |  14 +-
 .../tests/streaming/get_by_timestamp.rs            |  14 +-
 core/sdk/Cargo.toml                                |   4 +
 core/sdk/src/clients/consumer.rs                   |   8 +-
 core/sdk/src/clients/consumer_builder.rs           |  13 +-
 core/sdk/src/clients/mod.rs                        |   6 +-
 core/sdk/src/clients/producer.rs                   | 526 +++++++++++----------
 core/sdk/src/clients/producer_builder.rs           |  79 ++--
 core/sdk/src/clients/producer_config.rs            | 143 ++++++
 core/sdk/src/clients/producer_dispatcher.rs        | 464 ++++++++++++++++++
 core/sdk/src/clients/producer_error_callback.rs    |  66 +++
 core/sdk/src/clients/producer_sharding.rs          | 430 +++++++++++++++++
 core/sdk/src/prelude.rs                            |   1 +
 .../stream_builder/build/build_iggy_consumer.rs    |   4 +-
 .../stream_builder/build/build_iggy_producer.rs    |  15 +-
 .../stream_builder/config/config_iggy_consumer.rs  |  32 +-
 .../stream_builder/config/config_iggy_producer.rs  |  74 ++-
 .../stream_builder/config/config_iggy_stream.rs    |  30 +-
 39 files changed, 2097 insertions(+), 435 deletions(-)
 copy core/{examples/src/lib.rs => integration/tests/sdk/mod.rs} (98%)
 create mode 100644 core/integration/tests/sdk/producer/background.rs
 create mode 100644 core/integration/tests/sdk/producer/mod.rs
 create mode 100644 core/sdk/src/clients/producer_config.rs
 create mode 100644 core/sdk/src/clients/producer_dispatcher.rs
 create mode 100644 core/sdk/src/clients/producer_error_callback.rs
 create mode 100644 core/sdk/src/clients/producer_sharding.rs


(iggy) 01/01: add initial version of connectors http api

Posted by pi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

piotr pushed a commit to branch connectors_api
in repository https://gitbox.apache.org/repos/asf/iggy.git

commit 238287508a91d64e0b573634809356b9ebf29477
Author: spetz <pi...@gmail.com>
AuthorDate: Tue Jun 10 20:31:04 2025 +0200

    add initial version of connectors http api
---
 Cargo.lock                                    |   4 +
 Cargo.toml                                    |   1 +
 core/connectors/README.md                     |   3 +-
 core/connectors/runtime/Cargo.toml            |   4 +
 core/connectors/runtime/README.md             |  22 ++++++
 core/connectors/runtime/config.toml           |   7 ++
 core/connectors/runtime/src/api/auth.rs       |  45 +++++++++++
 core/connectors/runtime/src/api/config.rs     |  62 ++++++++++++++++
 core/connectors/runtime/src/api/error.rs      |  74 ++++++++++++++++++
 core/connectors/runtime/src/api/mod.rs        |  69 +++++++++++++++++
 core/connectors/runtime/src/api/models.rs     | 103 ++++++++++++++++++++++++++
 core/connectors/runtime/src/api/sink.rs       |  89 ++++++++++++++++++++++
 core/connectors/runtime/src/api/source.rs     |  89 ++++++++++++++++++++++
 core/connectors/runtime/src/configs.rs        |  44 ++++++++---
 core/connectors/runtime/src/context.rs        | 101 +++++++++++++++++++++++++
 core/connectors/runtime/src/error.rs          |  20 +++++
 core/connectors/runtime/src/main.rs           |  30 ++++++--
 core/connectors/runtime/src/manager/mod.rs    |  20 +++++
 core/connectors/runtime/src/manager/sink.rs   |  65 ++++++++++++++++
 core/connectors/runtime/src/manager/source.rs |  65 ++++++++++++++++
 core/connectors/runtime/src/sink.rs           |   8 ++
 core/connectors/runtime/src/source.rs         |   8 ++
 core/connectors/sinks/README.md               |   2 +
 core/connectors/sources/README.md             |   2 +
 24 files changed, 918 insertions(+), 19 deletions(-)

diff --git a/Cargo.lock b/Cargo.lock
index 76292c4b..910f342a 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -3859,6 +3859,7 @@ dependencies = [
 name = "iggy_connector_runtime"
 version = "0.1.0"
 dependencies = [
+ "axum 0.8.4",
  "config",
  "dashmap",
  "dlopen2",
@@ -3871,8 +3872,11 @@ dependencies = [
  "postcard",
  "serde",
  "serde_json",
+ "serde_yml",
+ "strum",
  "thiserror 2.0.12",
  "tokio",
+ "toml",
  "tracing",
  "tracing-subscriber",
 ]
diff --git a/Cargo.toml b/Cargo.toml
index 61e69b25..db00bf9c 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -76,6 +76,7 @@ bytes = "1.10.1"
 charming = "0.4.0"
 chrono = { version = "0.4.41", features = ["serde"] }
 clap = { version = "4.5.37", features = ["derive"] }
+common_macros = { version = "0.1.1", default-features = false }
 config = { version = "0.15.11" }
 comfy-table = "7.1.4"
 crc32fast = "1.4.2"
diff --git a/core/connectors/README.md b/core/connectors/README.md
index 68051e50..a9a1b319 100644
--- a/core/connectors/README.md
+++ b/core/connectors/README.md
@@ -25,7 +25,8 @@ The highly performant and modular runtime for statically typed, yet dynamically
 4. Start the Iggy server and invoke the following commands via Iggy CLI to create the example streams and topics used by the sample connectors.
 
 ```
-iggy --username iggy --password iggy stream create example
+iggy --username iggy --password iggy stream create example_stream
+iggy --username iggy --password iggy topic create example_stream example_topic 1 none 1d
 iggy --username iggy --password iggy stream create qw
 iggy --username iggy --password iggy topic create qw records 1 none 1d
 ```
diff --git a/core/connectors/runtime/Cargo.toml b/core/connectors/runtime/Cargo.toml
index a975d0ae..8fc6fba5 100644
--- a/core/connectors/runtime/Cargo.toml
+++ b/core/connectors/runtime/Cargo.toml
@@ -29,6 +29,7 @@ repository = "https://github.com/apache/iggy"
 readme = "../../README.md"
 
 [dependencies]
+axum = { workspace = true }
 config = { workspace = true }
 dashmap = { workspace = true }
 dlopen2 = { workspace = true }
@@ -41,7 +42,10 @@ once_cell = { workspace = true }
 postcard = { workspace = true }
 serde = { workspace = true }
 serde_json = { workspace = true }
+serde_yml = { workspace = true }
+strum = { workspace = true }
 thiserror = { workspace = true }
 tokio = { workspace = true }
+toml = { workspace = true }
 tracing = { workspace = true }
 tracing-subscriber = { workspace = true }
diff --git a/core/connectors/runtime/README.md b/core/connectors/runtime/README.md
index 23a35c11..3947fd31 100644
--- a/core/connectors/runtime/README.md
+++ b/core/connectors/runtime/README.md
@@ -23,3 +23,25 @@ password = "iggy"
 All the other config sections start either with `sources` or `sinks` depending on the connector type.
 
 Keep in mind that either of `toml`, `yaml`, or `json` formats are supported for the configuration file. The path to the configuration can be overriden by `IGGY_CONNECTORS_RUNTIME_CONFIG_PATH` environment variable. Each configuration section can be also additionally updated by using the following convention `IGGY_CONNECTORS_SECTION_NAME.KEY_NAME` e.g. `IGGY_CONNECTORS_IGGY_USERNAME` and so on.
+
+## HTTP API
+
+Connector runtime has an optional HTTP API that can be enabled by setting the `enabled` flag to `true` in the `[http_api]` section.
+
+```toml
+[http_api]
+enabled = true
+address = "127.0.0.1:8081"
+# api_key = "secret" # Optional API key for authentication to be passed as `api-key` header
+```
+
+Currently, it does expose the following endpoints:
+
+- `GET /`: welcome message.
+- `GET /health`: health status of the runtime.
+- `GET /sinks`: list of sinks.
+- `GET /sinks/{key}`: sink details.
+- `GET /sinks/{key}/config`: sink config, including the optional `format` query parametr to specify the config foramt.
+- `GET /sources`: list of sources.
+- `GET /sources/{key}`: source details.
+- `GET /sources/{key}/config`: source config, including the optional `format` query parameter to specify the config format.
diff --git a/core/connectors/runtime/config.toml b/core/connectors/runtime/config.toml
index d7361472..ff547853 100644
--- a/core/connectors/runtime/config.toml
+++ b/core/connectors/runtime/config.toml
@@ -15,6 +15,11 @@
 # specific language governing permissions and limitations
 # under the License.
 
+[http_api]
+enabled = true
+address = "127.0.0.1:8081"
+# api_key = "secret" # Optional API key for authentication to be passed as `api-key` header
+
 [iggy]
 address = "localhost:8090"
 username = "iggy"
@@ -45,6 +50,7 @@ value.static = "hello"
 enabled = true
 name = "Random source"
 path = "target/release/libiggy_connector_random_source"
+config_format = "json"
 
 [[sources.random.streams]]
 stream = "example"
@@ -70,6 +76,7 @@ value.static = "hello!"
 enabled = true
 name = "Quickwit sink 1"
 path = "target/release/libiggy_connector_quickwit_sink"
+config_format = "yaml"
 
 [[sinks.quickwit.streams]]
 stream = "qw"
diff --git a/core/connectors/runtime/src/api/auth.rs b/core/connectors/runtime/src/api/auth.rs
new file mode 100644
index 00000000..5d785819
--- /dev/null
+++ b/core/connectors/runtime/src/api/auth.rs
@@ -0,0 +1,45 @@
+use crate::context::RuntimeContext;
+use axum::body::Body;
+use axum::extract::State;
+use axum::{
+    http::{Request, StatusCode},
+    middleware::Next,
+    response::Response,
+};
+use std::sync::Arc;
+
+const API_KEY_HEADER: &str = "api-key";
+
+const PUBLIC_PATHS: &[&str] = &["/", "/health"];
+
+pub async fn resolve_api_key(
+    State(context): State<Arc<RuntimeContext>>,
+    request: Request<Body>,
+    next: Next,
+) -> Result<Response, StatusCode> {
+    if PUBLIC_PATHS.contains(&request.uri().path()) {
+        return Ok(next.run(request).await);
+    }
+
+    let Some(expected_api_key) = context.api_key.as_ref() else {
+        return Ok(next.run(request).await);
+    };
+
+    if expected_api_key.is_empty() {
+        return Ok(next.run(request).await);
+    }
+
+    let Some(api_key) = request
+        .headers()
+        .get(API_KEY_HEADER)
+        .and_then(|value| value.to_str().ok())
+    else {
+        return Err(StatusCode::UNAUTHORIZED);
+    };
+
+    if api_key != expected_api_key {
+        return Err(StatusCode::UNAUTHORIZED);
+    }
+
+    Ok(next.run(request).await)
+}
diff --git a/core/connectors/runtime/src/api/config.rs b/core/connectors/runtime/src/api/config.rs
new file mode 100644
index 00000000..474365d9
--- /dev/null
+++ b/core/connectors/runtime/src/api/config.rs
@@ -0,0 +1,62 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::{configs::ConfigFormat, error::RuntimeError};
+use axum::http::HeaderValue;
+use serde::{Deserialize, Serialize};
+use tracing::error;
+
+pub const JSON_HEADER: HeaderValue = HeaderValue::from_static("application/json");
+pub const YAML_HEADER: HeaderValue = HeaderValue::from_static("application/yaml");
+pub const TOML_HEADER: HeaderValue = HeaderValue::from_static("application/toml");
+pub const TEXT_HEADER: HeaderValue = HeaderValue::from_static("text/plain");
+
+#[derive(Debug, Clone, Deserialize, Serialize)]
+pub struct HttpApiConfig {
+    pub enabled: bool,
+    pub address: String,
+    pub api_key: Option<String>,
+}
+
+pub fn map_connector_config(
+    config: &serde_json::Value,
+    format: ConfigFormat,
+) -> Result<(HeaderValue, String), RuntimeError> {
+    match format {
+        ConfigFormat::Json => Ok((JSON_HEADER, config.to_string())),
+        ConfigFormat::Yaml => {
+            let config = serde_yml::to_value(config).map_err(|error| {
+                error!("Failed to convert configuration to YAML. {error}");
+                RuntimeError::CannotConvertConfiguration
+            })?;
+            let config = serde_yml::to_string(&config).map_err(|error| {
+                error!("Failed to serialize YAML configuration. {error}");
+                RuntimeError::CannotConvertConfiguration
+            })?;
+            Ok((YAML_HEADER, config))
+        }
+        ConfigFormat::Toml => {
+            let config = toml::to_string(config).map_err(|error| {
+                error!("Failed to convert configuration to TOML. {error}");
+                RuntimeError::CannotConvertConfiguration
+            })?;
+            Ok((TOML_HEADER, config))
+        }
+        ConfigFormat::Text => Ok((TEXT_HEADER, config.to_string())),
+    }
+}
diff --git a/core/connectors/runtime/src/api/error.rs b/core/connectors/runtime/src/api/error.rs
new file mode 100644
index 00000000..b14d301a
--- /dev/null
+++ b/core/connectors/runtime/src/api/error.rs
@@ -0,0 +1,74 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use axum::{Json, http::StatusCode, response::IntoResponse};
+use serde::Serialize;
+use thiserror::Error;
+use tracing::error;
+
+use crate::error::RuntimeError;
+
+#[derive(Debug, Error)]
+pub enum ApiError {
+    #[error(transparent)]
+    Error(#[from] RuntimeError),
+    #[error(transparent)]
+    JsonError(#[from] serde_json::Error),
+}
+
+#[derive(Debug, Serialize)]
+pub struct ErrorResponse {
+    pub code: String,
+    pub reason: String,
+}
+
+impl IntoResponse for ApiError {
+    fn into_response(self) -> axum::response::Response {
+        match self {
+            ApiError::Error(error) => {
+                error!("There was an error: {error}");
+                let status_code = match error {
+                    RuntimeError::MissingIggyCredentials => StatusCode::BAD_REQUEST,
+                    RuntimeError::InvalidConfiguration(_) => StatusCode::BAD_REQUEST,
+                    RuntimeError::CannotConvertConfiguration => StatusCode::BAD_REQUEST,
+                    RuntimeError::SinkNotFound(_) => StatusCode::NOT_FOUND,
+                    RuntimeError::SourceNotFound(_) => StatusCode::NOT_FOUND,
+                    _ => StatusCode::INTERNAL_SERVER_ERROR,
+                };
+                (
+                    status_code,
+                    Json(ErrorResponse {
+                        code: error.as_code().to_owned(),
+                        reason: error.to_string(),
+                    }),
+                )
+            }
+            ApiError::JsonError(error) => {
+                error!("There was a JSON error: {error}");
+                (
+                    StatusCode::INTERNAL_SERVER_ERROR,
+                    Json(ErrorResponse {
+                        code: "json_error".to_owned(),
+                        reason: error.to_string(),
+                    }),
+                )
+            }
+        }
+        .into_response()
+    }
+}
diff --git a/core/connectors/runtime/src/api/mod.rs b/core/connectors/runtime/src/api/mod.rs
new file mode 100644
index 00000000..8dc1317e
--- /dev/null
+++ b/core/connectors/runtime/src/api/mod.rs
@@ -0,0 +1,69 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use auth::resolve_api_key;
+use axum::{Json, Router, middleware, routing::get};
+use config::HttpApiConfig;
+use std::sync::Arc;
+
+use tokio::{net::TcpListener, spawn};
+use tracing::info;
+
+use crate::context::RuntimeContext;
+
+mod auth;
+pub mod config;
+mod error;
+mod models;
+mod sink;
+mod source;
+
+const NAME: &str = env!("CARGO_PKG_NAME");
+
+pub async fn init(config: &HttpApiConfig, context: Arc<RuntimeContext>) {
+    if !config.enabled {
+        info!("{NAME} HTTP API is disabled");
+        return;
+    }
+
+    let address = config.address.to_owned();
+    let mut app = Router::new()
+        .route("/", get(|| async { "Connector Runtime API" }))
+        .route(
+            "/health",
+            get(|| async { Json(serde_json::json!({ "status": "healthy" })) }),
+        )
+        .merge(sink::router(context.clone()))
+        .merge(source::router(context.clone()));
+
+    app = app.layer(middleware::from_fn_with_state(
+        context.clone(),
+        resolve_api_key,
+    ));
+
+    let Ok(listener) = TcpListener::bind(&address).await else {
+        panic!("{NAME} failed to bind to HTTP API address: {address}");
+    };
+
+    spawn(async move {
+        info!("Started {NAME} HTTP API on: {address}");
+        if let Err(error) = axum::serve(listener, app.into_make_service()).await {
+            panic!("Failed to start {NAME} HTTP API on: {address}. {error}");
+        }
+    });
+}
diff --git a/core/connectors/runtime/src/api/models.rs b/core/connectors/runtime/src/api/models.rs
new file mode 100644
index 00000000..12b4c7ed
--- /dev/null
+++ b/core/connectors/runtime/src/api/models.rs
@@ -0,0 +1,103 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use serde::{Deserialize, Serialize};
+
+use crate::{
+    configs::ConfigFormat,
+    manager::{
+        sink::{SinkDetails, SinkInfo},
+        source::{SourceDetails, SourceInfo},
+    },
+};
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct SinkInfoResponse {
+    pub id: u32,
+    pub key: String,
+    pub name: String,
+    pub path: String,
+    pub enabled: bool,
+    pub running: bool,
+    pub config_format: Option<ConfigFormat>,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct SinkDetailsResponse {
+    #[serde(flatten)]
+    pub info: SinkInfoResponse,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct SourceInfoResponse {
+    pub id: u32,
+    pub key: String,
+    pub name: String,
+    pub path: String,
+    pub enabled: bool,
+    pub running: bool,
+    pub config_format: Option<ConfigFormat>,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct SourceDetailsResponse {
+    #[serde(flatten)]
+    pub info: SourceInfoResponse,
+}
+
+impl From<&SinkInfo> for SinkInfoResponse {
+    fn from(sink: &SinkInfo) -> Self {
+        SinkInfoResponse {
+            id: sink.id,
+            key: sink.key.to_owned(),
+            name: sink.name.to_owned(),
+            path: sink.path.to_owned(),
+            enabled: sink.enabled,
+            running: sink.running,
+            config_format: sink.config_format,
+        }
+    }
+}
+
+impl From<&SinkDetails> for SinkDetailsResponse {
+    fn from(sink: &SinkDetails) -> Self {
+        let info = &sink.info;
+        SinkDetailsResponse { info: info.into() }
+    }
+}
+
+impl From<&SourceInfo> for SourceInfoResponse {
+    fn from(source: &SourceInfo) -> Self {
+        SourceInfoResponse {
+            id: source.id,
+            key: source.key.to_owned(),
+            name: source.name.to_owned(),
+            path: source.path.to_owned(),
+            enabled: source.enabled,
+            running: source.running,
+            config_format: source.config_format,
+        }
+    }
+}
+
+impl From<&SourceDetails> for SourceDetailsResponse {
+    fn from(source: &SourceDetails) -> Self {
+        let info = &source.info;
+        SourceDetailsResponse { info: info.into() }
+    }
+}
diff --git a/core/connectors/runtime/src/api/sink.rs b/core/connectors/runtime/src/api/sink.rs
new file mode 100644
index 00000000..f9994b6e
--- /dev/null
+++ b/core/connectors/runtime/src/api/sink.rs
@@ -0,0 +1,89 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use super::{
+    config::map_connector_config,
+    error::ApiError,
+    models::{SinkDetailsResponse, SinkInfoResponse},
+};
+use crate::{configs::ConfigFormat, context::RuntimeContext, error::RuntimeError};
+use axum::{
+    Json, Router,
+    extract::{Path, Query, State},
+    http::{HeaderMap, StatusCode, header},
+    response::IntoResponse,
+    routing::get,
+};
+use serde::Deserialize;
+use std::sync::Arc;
+
+pub fn router(state: Arc<RuntimeContext>) -> Router {
+    Router::new()
+        .route("/sinks", get(get_sinks))
+        .route("/sinks/{key}", get(get_sink))
+        .route("/sinks/{key}/config", get(get_sink_config))
+        .with_state(state)
+}
+
+async fn get_sinks(
+    State(context): State<Arc<RuntimeContext>>,
+) -> Result<Json<Vec<SinkInfoResponse>>, ApiError> {
+    let sinks = context
+        .sinks
+        .get_all()
+        .into_values()
+        .map(|sink| sink.into())
+        .collect::<Vec<_>>();
+    Ok(Json(sinks))
+}
+
+async fn get_sink(
+    State(context): State<Arc<RuntimeContext>>,
+    Path(key): Path<String>,
+) -> Result<Json<SinkDetailsResponse>, ApiError> {
+    let Some(sink) = context.sinks.get(&key) else {
+        return Err(ApiError::Error(RuntimeError::SinkNotFound(key)));
+    };
+    Ok(Json(sink.into()))
+}
+
+async fn get_sink_config(
+    State(context): State<Arc<RuntimeContext>>,
+    Path(key): Path<String>,
+    Query(query): Query<GetSinkConfig>,
+) -> Result<impl IntoResponse, ApiError> {
+    let Some(sink) = context.sinks.get(&key) else {
+        return Err(ApiError::Error(RuntimeError::SinkNotFound(key)));
+    };
+    let Some(config) = sink.config.clone() else {
+        return Ok(StatusCode::NOT_FOUND.into_response());
+    };
+
+    let format = query
+        .format
+        .unwrap_or(sink.info.config_format.unwrap_or_default());
+    let (content_type, config) = map_connector_config(&config, format)?;
+    let mut headers = HeaderMap::new();
+    headers.insert(header::CONTENT_TYPE, content_type);
+    Ok((headers, config).into_response())
+}
+
+#[derive(Debug, Deserialize)]
+struct GetSinkConfig {
+    format: Option<ConfigFormat>,
+}
diff --git a/core/connectors/runtime/src/api/source.rs b/core/connectors/runtime/src/api/source.rs
new file mode 100644
index 00000000..55a69eb8
--- /dev/null
+++ b/core/connectors/runtime/src/api/source.rs
@@ -0,0 +1,89 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use super::{
+    config::map_connector_config,
+    error::ApiError,
+    models::{SourceDetailsResponse, SourceInfoResponse},
+};
+use crate::{configs::ConfigFormat, context::RuntimeContext, error::RuntimeError};
+use axum::{
+    Json, Router,
+    extract::{Path, Query, State},
+    http::{HeaderMap, StatusCode, header},
+    response::IntoResponse,
+    routing::get,
+};
+use serde::Deserialize;
+use std::sync::Arc;
+
+pub fn router(state: Arc<RuntimeContext>) -> Router {
+    Router::new()
+        .route("/sources", get(get_sources))
+        .route("/sources/{key}", get(get_source))
+        .route("/sources/{key}/config", get(get_source_config))
+        .with_state(state)
+}
+
+async fn get_sources(
+    State(context): State<Arc<RuntimeContext>>,
+) -> Result<Json<Vec<SourceInfoResponse>>, ApiError> {
+    let sources = context
+        .sources
+        .get_all()
+        .into_values()
+        .map(|source| source.into())
+        .collect::<Vec<_>>();
+    Ok(Json(sources))
+}
+
+async fn get_source(
+    State(context): State<Arc<RuntimeContext>>,
+    Path(key): Path<String>,
+) -> Result<Json<SourceDetailsResponse>, ApiError> {
+    let Some(source) = context.sources.get(&key) else {
+        return Err(ApiError::Error(RuntimeError::SourceNotFound(key)));
+    };
+    Ok(Json(source.into()))
+}
+
+async fn get_source_config(
+    State(context): State<Arc<RuntimeContext>>,
+    Path(key): Path<String>,
+    Query(query): Query<GetSourceConfig>,
+) -> Result<impl IntoResponse, ApiError> {
+    let Some(source) = context.sources.get(&key) else {
+        return Err(ApiError::Error(RuntimeError::SourceNotFound(key)));
+    };
+    let Some(config) = source.config.clone() else {
+        return Ok(StatusCode::NOT_FOUND.into_response());
+    };
+
+    let format = query
+        .format
+        .unwrap_or(source.info.config_format.unwrap_or_default());
+    let (content_type, config) = map_connector_config(&config, format)?;
+    let mut headers = HeaderMap::new();
+    headers.insert(header::CONTENT_TYPE, content_type);
+    Ok((headers, config).into_response())
+}
+
+#[derive(Debug, Deserialize)]
+struct GetSourceConfig {
+    format: Option<ConfigFormat>,
+}
diff --git a/core/connectors/runtime/src/configs.rs b/core/connectors/runtime/src/configs.rs
index aafc9a48..9b940d10 100644
--- a/core/connectors/runtime/src/configs.rs
+++ b/core/connectors/runtime/src/configs.rs
@@ -18,16 +18,36 @@
 
 use iggy_connector_sdk::{Schema, transforms::TransformType};
 use serde::{Deserialize, Serialize};
-use std::collections::HashMap;
+use std::{collections::HashMap, sync::Arc};
+use strum::Display;
 
-#[derive(Debug, Deserialize, Serialize)]
+use crate::api::config::HttpApiConfig;
+
+#[derive(
+    Debug, Default, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Deserialize, Serialize, Display,
+)]
+#[serde(rename_all = "lowercase")]
+pub enum ConfigFormat {
+    #[strum(to_string = "json")]
+    Json,
+    #[strum(to_string = "yaml")]
+    Yaml,
+    #[strum(to_string = "toml")]
+    Toml,
+    #[default]
+    #[strum(to_string = "text")]
+    Text,
+}
+
+#[derive(Debug, Clone, Deserialize, Serialize)]
 pub struct RuntimeConfig {
+    pub http_api: HttpApiConfig,
     pub iggy: IggyConfig,
     pub sinks: HashMap<String, SinkConfig>,
     pub sources: HashMap<String, SourceConfig>,
 }
 
-#[derive(Debug, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize)]
 pub struct IggyConfig {
     pub address: String,
     pub username: Option<String>,
@@ -35,17 +55,18 @@ pub struct IggyConfig {
     pub token: Option<String>,
 }
 
-#[derive(Debug, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize)]
 pub struct SinkConfig {
     pub enabled: bool,
     pub name: String,
     pub path: String,
     pub transforms: Option<TransformsConfig>,
     pub streams: Vec<StreamConsumerConfig>,
-    pub config: Option<serde_json::Value>,
+    pub config_format: Option<ConfigFormat>,
+    pub config: Option<Arc<serde_json::Value>>,
 }
 
-#[derive(Debug, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize)]
 pub struct StreamConsumerConfig {
     pub stream: String,
     pub topics: Vec<String>,
@@ -55,7 +76,7 @@ pub struct StreamConsumerConfig {
     pub consumer_group: Option<String>,
 }
 
-#[derive(Debug, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize)]
 pub struct StreamProducerConfig {
     pub stream: String,
     pub topic: String,
@@ -64,23 +85,24 @@ pub struct StreamProducerConfig {
     pub linger_time: Option<String>,
 }
 
-#[derive(Debug, Serialize, Deserialize)]
+#[derive(Debug, Clone, Serialize, Deserialize)]
 pub struct SourceConfig {
     pub enabled: bool,
     pub name: String,
     pub path: String,
     pub transforms: Option<TransformsConfig>,
     pub streams: Vec<StreamProducerConfig>,
-    pub config: Option<serde_json::Value>,
+    pub config_format: Option<ConfigFormat>,
+    pub config: Option<Arc<serde_json::Value>>,
 }
 
-#[derive(Debug, Default, Serialize, Deserialize)]
+#[derive(Debug, Clone, Default, Serialize, Deserialize)]
 pub struct TransformsConfig {
     #[serde(flatten)]
     pub transforms: HashMap<TransformType, serde_json::Value>,
 }
 
-#[derive(Debug, Default, Serialize, Deserialize)]
+#[derive(Debug, Clone, Default, Serialize, Deserialize)]
 pub struct SharedTransformConfig {
     pub enabled: bool,
 }
diff --git a/core/connectors/runtime/src/context.rs b/core/connectors/runtime/src/context.rs
new file mode 100644
index 00000000..e962c608
--- /dev/null
+++ b/core/connectors/runtime/src/context.rs
@@ -0,0 +1,101 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use crate::{
+    SinkConnectorWrapper, SourceConnectorWrapper,
+    configs::RuntimeConfig,
+    manager::{
+        sink::{SinkDetails, SinkInfo, SinkManager},
+        source::{SourceDetails, SourceInfo, SourceManager},
+    },
+};
+
+use tracing::error;
+
+pub struct RuntimeContext {
+    pub sinks: SinkManager,
+    pub sources: SourceManager,
+    pub api_key: Option<String>,
+}
+
+pub fn init(
+    config: &RuntimeConfig,
+    sink_wrappers: &[SinkConnectorWrapper],
+    source_wrappers: &[SourceConnectorWrapper],
+) -> RuntimeContext {
+    RuntimeContext {
+        sinks: SinkManager::new(map_sinks(config, sink_wrappers)),
+        sources: SourceManager::new(map_sources(config, source_wrappers)),
+        api_key: config.http_api.api_key.clone(),
+    }
+}
+
+fn map_sinks(config: &RuntimeConfig, sink_wrappers: &[SinkConnectorWrapper]) -> Vec<SinkDetails> {
+    let mut sinks = vec![];
+    for sink_wrapper in sink_wrappers.iter() {
+        for sink_plugin in sink_wrapper.plugins.iter() {
+            let Some(sink_config) = config.sinks.get(&sink_plugin.key) else {
+                error!("Missing sink config for: {}", sink_plugin.key);
+                continue;
+            };
+
+            sinks.push(SinkDetails {
+                info: SinkInfo {
+                    id: sink_plugin.id,
+                    key: sink_plugin.key.to_owned(),
+                    name: sink_plugin.name.to_owned(),
+                    path: sink_plugin.path.to_owned(),
+                    enabled: sink_config.enabled,
+                    running: sink_config.enabled,
+                    config_format: sink_plugin.config_format,
+                },
+                config: sink_config.config.clone(),
+            });
+        }
+    }
+    sinks
+}
+
+fn map_sources(
+    config: &RuntimeConfig,
+    source_wrappers: &[SourceConnectorWrapper],
+) -> Vec<SourceDetails> {
+    let mut sources = vec![];
+    for source_wrapper in source_wrappers.iter() {
+        for source_plugin in source_wrapper.plugins.iter() {
+            let Some(source_config) = config.sources.get(&source_plugin.key) else {
+                error!("Missing source config for: {}", source_plugin.key);
+                continue;
+            };
+
+            sources.push(SourceDetails {
+                info: SourceInfo {
+                    id: source_plugin.id,
+                    key: source_plugin.key.to_owned(),
+                    name: source_plugin.name.to_owned(),
+                    path: source_plugin.path.to_owned(),
+                    enabled: source_config.enabled,
+                    running: source_config.enabled,
+                    config_format: source_plugin.config_format,
+                },
+                config: source_config.config.clone(),
+            });
+        }
+    }
+    sources
+}
diff --git a/core/connectors/runtime/src/error.rs b/core/connectors/runtime/src/error.rs
index 702353e8..c622feab 100644
--- a/core/connectors/runtime/src/error.rs
+++ b/core/connectors/runtime/src/error.rs
@@ -38,4 +38,24 @@ pub enum RuntimeError {
     IggyError(#[from] iggy::prelude::IggyError),
     #[error("Missing Iggy credentials")]
     MissingIggyCredentials,
+    #[error("JSON error")]
+    JsonError(#[from] serde_json::Error),
+    #[error("Sink not found with key: {0}")]
+    SinkNotFound(String),
+    #[error("Source not found with key: {0}")]
+    SourceNotFound(String),
+    #[error("Cannot convert configuration")]
+    CannotConvertConfiguration,
+}
+
+impl RuntimeError {
+    pub fn as_code(&self) -> &'static str {
+        match self {
+            RuntimeError::SinkNotFound(_) => "sink_not_found",
+            RuntimeError::SourceNotFound(_) => "source_not_found",
+            RuntimeError::MissingIggyCredentials => "invalid_configuration",
+            RuntimeError::InvalidConfiguration(_) => "invalid_configuration",
+            _ => "error",
+        }
+    }
 }
diff --git a/core/connectors/runtime/src/main.rs b/core/connectors/runtime/src/main.rs
index f87057bf..67bf303a 100644
--- a/core/connectors/runtime/src/main.rs
+++ b/core/connectors/runtime/src/main.rs
@@ -17,7 +17,7 @@
  */
 
 use config::{Config, Environment, File};
-use configs::RuntimeConfig;
+use configs::{ConfigFormat, RuntimeConfig};
 use dlopen2::wrapper::{Container, WrapperApi};
 use error::RuntimeError;
 use iggy::prelude::{Client, IggyClient, IggyClientBuilder, IggyConsumer, IggyProducer};
@@ -36,8 +36,11 @@ use std::{
 use tracing::{debug, error, info};
 use tracing_subscriber::{EnvFilter, Registry, layer::SubscriberExt, util::SubscriberInitExt};
 
+mod api;
 pub(crate) mod configs;
+pub(crate) mod context;
 pub(crate) mod error;
+mod manager;
 mod sink;
 mod source;
 mod transform;
@@ -91,10 +94,11 @@ async fn main() -> Result<(), RuntimeError> {
         .try_deserialize()
         .expect("Failed to deserialize runtime config");
 
-    let iggy_address = config.iggy.address;
-    let iggy_username = config.iggy.username;
-    let iggy_password = config.iggy.password;
-    let iggy_token = config.iggy.token;
+    let iggy_config = config.iggy.clone();
+    let iggy_address = iggy_config.address;
+    let iggy_username = iggy_config.username;
+    let iggy_password = iggy_config.password;
+    let iggy_token = iggy_config.token;
     let consumer_client = create_iggy_client(
         &iggy_address,
         iggy_username.as_deref(),
@@ -112,8 +116,8 @@ async fn main() -> Result<(), RuntimeError> {
     .await?;
     producer_client.connect().await?;
 
-    let sources = source::init(config.sources, &producer_client).await?;
-    let sinks = sink::init(config.sinks, &consumer_client).await?;
+    let sources = source::init(config.sources.clone(), &producer_client).await?;
+    let sinks = sink::init(config.sinks.clone(), &consumer_client).await?;
 
     let mut sink_wrappers = vec![];
     let mut sink_with_plugins = HashMap::new();
@@ -149,6 +153,10 @@ async fn main() -> Result<(), RuntimeError> {
         );
     }
 
+    let context = context::init(&config, &sink_wrappers, &source_wrappers);
+    let context = Arc::new(context);
+    api::init(&config.http_api, context).await;
+
     source::handle(source_wrappers);
     sink::consume(sink_wrappers);
     info!("All sources and sinks spawned.");
@@ -261,6 +269,10 @@ struct SinkConnector {
 
 struct SinkConnectorPlugin {
     id: u32,
+    key: String,
+    name: String,
+    path: String,
+    config_format: Option<ConfigFormat>,
     consumers: Vec<SinkConnectorConsumer>,
 }
 
@@ -288,6 +300,10 @@ struct SourceConnector {
 
 struct SourceConnectorPlugin {
     id: u32,
+    key: String,
+    name: String,
+    path: String,
+    config_format: Option<ConfigFormat>,
     transforms: Vec<Arc<dyn Transform>>,
     producer: Option<SourceConnectorProducer>,
 }
diff --git a/core/connectors/runtime/src/manager/mod.rs b/core/connectors/runtime/src/manager/mod.rs
new file mode 100644
index 00000000..8efc27b1
--- /dev/null
+++ b/core/connectors/runtime/src/manager/mod.rs
@@ -0,0 +1,20 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+pub mod sink;
+pub mod source;
diff --git a/core/connectors/runtime/src/manager/sink.rs b/core/connectors/runtime/src/manager/sink.rs
new file mode 100644
index 00000000..10e1282a
--- /dev/null
+++ b/core/connectors/runtime/src/manager/sink.rs
@@ -0,0 +1,65 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use std::{collections::HashMap, sync::Arc};
+
+use crate::configs::ConfigFormat;
+
+#[derive(Debug)]
+pub struct SinkManager {
+    sinks: HashMap<String, SinkDetails>,
+}
+
+impl SinkManager {
+    pub fn new(sinks: Vec<SinkDetails>) -> Self {
+        Self {
+            sinks: sinks
+                .into_iter()
+                .map(|sink| (sink.info.key.to_owned(), sink))
+                .collect(),
+        }
+    }
+
+    pub fn get(&self, key: &str) -> Option<&SinkDetails> {
+        self.sinks.get(key)
+    }
+
+    pub fn get_all(&self) -> HashMap<&String, &SinkInfo> {
+        self.sinks
+            .iter()
+            .map(|(key, sink)| (key, &sink.info))
+            .collect()
+    }
+}
+
+#[derive(Debug)]
+pub struct SinkInfo {
+    pub id: u32,
+    pub key: String,
+    pub name: String,
+    pub path: String,
+    pub enabled: bool,
+    pub running: bool,
+    pub config_format: Option<ConfigFormat>,
+}
+
+#[derive(Debug)]
+pub struct SinkDetails {
+    pub info: SinkInfo,
+    pub config: Option<Arc<serde_json::Value>>,
+}
diff --git a/core/connectors/runtime/src/manager/source.rs b/core/connectors/runtime/src/manager/source.rs
new file mode 100644
index 00000000..5391dd1e
--- /dev/null
+++ b/core/connectors/runtime/src/manager/source.rs
@@ -0,0 +1,65 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use std::{collections::HashMap, sync::Arc};
+
+use crate::configs::ConfigFormat;
+
+#[derive(Debug)]
+pub struct SourceManager {
+    sources: HashMap<String, SourceDetails>,
+}
+
+impl SourceManager {
+    pub fn new(sources: Vec<SourceDetails>) -> Self {
+        Self {
+            sources: sources
+                .into_iter()
+                .map(|source| (source.info.key.to_owned(), source))
+                .collect(),
+        }
+    }
+
+    pub fn get(&self, key: &str) -> Option<&SourceDetails> {
+        self.sources.get(key)
+    }
+
+    pub fn get_all(&self) -> HashMap<&String, &SourceInfo> {
+        self.sources
+            .iter()
+            .map(|(key, source)| (key, &source.info))
+            .collect()
+    }
+}
+
+#[derive(Debug)]
+pub struct SourceInfo {
+    pub id: u32,
+    pub key: String,
+    pub name: String,
+    pub path: String,
+    pub enabled: bool,
+    pub running: bool,
+    pub config_format: Option<ConfigFormat>,
+}
+
+#[derive(Debug)]
+pub struct SourceDetails {
+    pub info: SourceInfo,
+    pub config: Option<Arc<serde_json::Value>>,
+}
diff --git a/core/connectors/runtime/src/sink.rs b/core/connectors/runtime/src/sink.rs
index 78008011..36fd631b 100644
--- a/core/connectors/runtime/src/sink.rs
+++ b/core/connectors/runtime/src/sink.rs
@@ -63,6 +63,10 @@ pub async fn init(
             );
             container.plugins.push(SinkConnectorPlugin {
                 id: plugin_id,
+                key: key.to_owned(),
+                name: name.to_owned(),
+                path: path.to_owned(),
+                config_format: config.config_format,
                 consumers: vec![],
             });
         } else {
@@ -76,6 +80,10 @@ pub async fn init(
                     container,
                     plugins: vec![SinkConnectorPlugin {
                         id: plugin_id,
+                        key: key.to_owned(),
+                        name: name.to_owned(),
+                        path: path.to_owned(),
+                        config_format: config.config_format,
                         consumers: vec![],
                     }],
                 },
diff --git a/core/connectors/runtime/src/source.rs b/core/connectors/runtime/src/source.rs
index b6cdeb4d..f1480c00 100644
--- a/core/connectors/runtime/src/source.rs
+++ b/core/connectors/runtime/src/source.rs
@@ -65,6 +65,10 @@ pub async fn init(
             );
             container.plugins.push(SourceConnectorPlugin {
                 id: plugin_id,
+                key: key.to_owned(),
+                name: name.to_owned(),
+                path: path.to_owned(),
+                config_format: config.config_format,
                 producer: None,
                 transforms: vec![],
             });
@@ -79,6 +83,10 @@ pub async fn init(
                     container,
                     plugins: vec![SourceConnectorPlugin {
                         id: plugin_id,
+                        key: key.to_owned(),
+                        name: name.to_owned(),
+                        path: path.to_owned(),
+                        config_format: config.config_format,
                         producer: None,
                         transforms: vec![],
                     }],
diff --git a/core/connectors/sinks/README.md b/core/connectors/sinks/README.md
index 78541e30..bf2dc6c7 100644
--- a/core/connectors/sinks/README.md
+++ b/core/connectors/sinks/README.md
@@ -37,6 +37,7 @@ pub struct SinkConfig {
     pub path: String,
     pub transforms: Option<TransformsConfig>,
     pub streams: Vec<StreamConsumerConfig>,
+    pub config_format: ConfigFormat,
     pub config: Option<serde_json::Value>,
 }
 ```
@@ -49,6 +50,7 @@ Below is the example configuration for a sink connector, using `stdout` as it's
 enabled = true
 name = "Stdout sink"
 path = "target/release/libiggy_connector_stdout_sink"
+config_format = "toml"
 
 # Collection of the streams from which messages are consumed
 [[sinks.stdout.streams]]
diff --git a/core/connectors/sources/README.md b/core/connectors/sources/README.md
index f4d39cae..8399e789 100644
--- a/core/connectors/sources/README.md
+++ b/core/connectors/sources/README.md
@@ -31,6 +31,7 @@ pub struct SourceConfig {
     pub path: String,
     pub transforms: Option<TransformsConfig>,
     pub streams: Vec<StreamProducerConfig>,
+    pub config_format: ConfigFormat,
     pub config: Option<serde_json::Value>,
 }
 ```
@@ -43,6 +44,7 @@ Below is the example configuration for a source connector, using `random` as it'
 enabled = true # Toggle source on/off
 name = "Random source" # Name of the source
 path = "libiggy_connector_random_source" # Path to the source connector
+config_format = "toml"
 
 # Collection of the streams to which the produced messages are sent
 [[sources.random.streams]]