[I] Split connectors (sinks/sources) configuration from the connectors runtime configuration [iggy]

["mmodzelewski (via GitHub)" <gi...@apache.org>]

mmodzelewski opened a new issue, #2318:
URL: https://github.com/apache/iggy/issues/2318

   ## Summary
   
   Split connector-specific configurations from the main runtime configuration file into separate per-connector configuration files organized in directories.
   
   ## Description
   
   The connectors runtime currently uses a single monolithic `config.toml` file that contains both runtime settings and all connector configurations. As the number of connectors grows, this approach becomes difficult to manage and maintain. This issue proposes refactoring the configuration structure to improve modularity and scalability.
   
   ## Proposed Changes
   
   ### Current Structure (Problem)
   - Single `config.toml` file containing:
     - Runtime settings (HTTP, Iggy client, state)
     - All sink configurations under `[sinks.<sink_name>]`
     - All source configurations under `[sources.<source_name>]`
   - Large, monolithic configuration file
   - Difficult to manage when multiple connectors are configured
   - Changes to one connector affect the entire configuration file
   
   ### Proposed Structure (Solution)
   - Main `config.toml` should contain only:
     - Runtime settings (HTTP, Iggy client, state)
     - Directory paths pointing to connector configurations:
   ```toml
   [sinks]
   config_dir = "path/to/sinks"
     
   [sources]
   config_dir = "path/to/sources"
   ```
   
   - Individual connector configs in separate files:
     - `sinks/*.toml` - One file per sink connector
     - `sources/*.toml` - One file per source connector
   
   ### Expected Benefits
   
   1. **Modularity**: Each connector will have its own isolated configuration file
   2. **Scalability**: Adding or removing connectors won't require modifying a central config
   3. **Organization**: Cleaner separation between runtime and connector-specific settings
   4. **Flexibility**: Per-connector environment variable support using `IGGY_SINK_<NAME>_*` and `IGGY_SOURCE_<NAME>_*` prefixes
   
   ### Configuration Format
   
   Individual connector files should contain only the connector-specific configuration at the top level, without the parent `[sinks.<name>]` or `[sources.<name>]` prefix. For example, a `sinks/stdout.toml` would look like:
   
   ```toml
   enabled = true
   name = "Stdout sink"
   path = "target/release/libiggy_connector_stdout_sink"
   
   [[streams]]
   stream = "example_stream"
   topics = ["example_topic"]
   # ...
   
   [config]
   print_payload = false
   ```
   
   
   ### Implementation Requirements
   
   - Load configurations dynamically from specified directories at runtime
   - Support graceful handling when `config_dir` is empty or directory doesn't exist
   - Support per-connector environment variable overrides
   - Provide example configurations in the new structure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Split connectors (sinks/sources) configuration from the connectors runtime configuration [iggy]

["mmodzelewski (via GitHub)" <gi...@apache.org>]

mmodzelewski closed issue #2318: Split connectors (sinks/sources) configuration from the connectors runtime configuration
URL: https://github.com/apache/iggy/issues/2318


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org