You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "jpatidar30 (via GitHub)" <gi...@apache.org> on 2026/02/10 11:43:24 UTC
[I] [Bug] Connector Runtime: IoError(Unsupported) on Linux Metadata and Recursive Plugin Loading Loop [iggy]
jpatidar30 opened a new issue, #2712:
URL: https://github.com/apache/iggy/issues/2712
### **Context**
I am currently working on a data pipeline to sync data from **Iggy** to an **Apache Iceberg** table via a REST catalog (Lakekeeper). During deployment on AWS, I encountered two critical blockers in the `iggy-connectors` runtime that prevent the sink from initializing.
---
### **Description of Blockers**
#### **1. Metadata Panic (`creation time is not available`)**
The `local_provider.rs` attempts to access file creation metadata via `std::fs::Metadata::created()`. On many Linux environments (including AWS EBS/XFS and Docker Overlay2), the filesystem driver does not support "birth time." This causes a hard panic when the runtime scans the `conf.d` directory.
**Observed Error:**
`Error: IoError(Error { kind: Unsupported, message: "creation time is not available on this platform currently" })`
**Temporary Workaround:** I had to patch the source to use `.modified()` as a fallback when `.created()` returns `Unsupported`.
### **2. Forced `.so` Plugin Loading for Internal Sinks**
Even after addressing metadata issues, the runtime fails to load the **internal** Iceberg sink when using a configuration file in `conf.d`.
The configuration parser requires the `path` field to be present in the TOML. However, the runtime interprets the presence of this `path` as a instruction to load an external shared library. It appends `.so` to the string and panics because it cannot find the file, even though the Iceberg driver is already compiled into the binary.
**I am using this configuration:**
```toml
version = 1
type = "sink"
key = "iceberg"
enabled = true
name = "iceberg-market-sink"
path = "iceberg_sink_state"
[[streams]]
stream = "1"
topics = ["1"]
group_id = "iggy_iceberg_group"
schema = "json"
[state]
path = "iceberg_sink_state"
[config]
catalog_type = "rest"
catalog_uri = ""
warehouse = ""
table_namespace = ""
table_name = ""
region = "ap-northeast-1"
s3_bucket = ""
commit_interval = "1m"
max_file_size_bytes = 134217728
[config.schema]
fields = [
{ name = "id", type = "long", nullable = false },
]
---
And I am getting this error:
thread 'main' (1) panicked at core/connectors/runtime/src/sink.rs:89:49: Failed to load sink container: OpeningLibraryError(Custom { kind: Other, error: "iceberg.so: cannot open shared object file: No such file or directory" })
### **System Information**
* **Iggy Version:** `v0.5.4` (and `main` as of Feb 2026)
* **OS:** Ubuntu 24.04 (AWS EC2)
* **Filesystem:** EBS (XFS/ext4)
* **Deployment:** Docker (privileged mode)
---
### **Steps to Reproduce**
1. Create a sink configuration in `/etc/iggy/connectors/conf.d/iceberg_sink.toml`.
2. Include the required `path` field (e.g., `path = "iceberg"`).
3. Set `key = "iceberg"` and `type = "sink"`.
4. Start the `iggy-connectors` runtime.
5. Observe the panic as the loader attempts to find an external `iceberg.so` file.
---
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Connector Runtime: IoError(Unsupported) on Linux Metadata and Recursive Plugin Loading Loop [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on issue #2712:
URL: https://github.com/apache/iggy/issues/2712#issuecomment-3880252941
@jpatidar30 please confirm that it works :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Connector Runtime: IoError(Unsupported) on Linux Metadata and Recursive Plugin Loading Loop [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on issue #2712:
URL: https://github.com/apache/iggy/issues/2712#issuecomment-4004206216
Closing due to inactivity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Connector Runtime: IoError(Unsupported) on Linux Metadata and Recursive Plugin Loading Loop [iggy]
Posted by "spetz (via GitHub)" <gi...@apache.org>.
spetz closed issue #2712: [Bug] Connector Runtime: IoError(Unsupported) on Linux Metadata and Recursive Plugin Loading Loop
URL: https://github.com/apache/iggy/issues/2712
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Connector Runtime: IoError(Unsupported) on Linux Metadata and Recursive Plugin Loading Loop [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on issue #2712:
URL: https://github.com/apache/iggy/issues/2712#issuecomment-3877708837
alternatively, you can check out our rolling release where we compile these libs and put them as assets. keep in mind that this should not be used in production: https://github.com/apache/iggy/releases/tag/edge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Connector Runtime: IoError(Unsupported) on Linux Metadata and Recursive Plugin Loading Loop [iggy]
Posted by "jpatidar30 (via GitHub)" <gi...@apache.org>.
jpatidar30 commented on issue #2712:
URL: https://github.com/apache/iggy/issues/2712#issuecomment-3877558244
Thanks for looking into this @hubcio
To answer your questions -
1. I am running iggy and iggy-connect docker images
2. No i dont have any plugins.
3. The one i shared in the config seems relative but I've tried absolute paths as well ended up with the same error.
4. Doesn't seem to exists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Connector Runtime: IoError(Unsupported) on Linux Metadata and Recursive Plugin Loading Loop [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on issue #2712:
URL: https://github.com/apache/iggy/issues/2712#issuecomment-3877109866
hey, i'll check this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Connector Runtime: IoError(Unsupported) on Linux Metadata and Recursive Plugin Loading Loop [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on issue #2712:
URL: https://github.com/apache/iggy/issues/2712#issuecomment-3877668904
thanks for the details @jpatidar30 , now it's clear whats going on.
the `apache/iggy-connect` docker image ships only the runtime binary, not the connector plugin `.so` files. you need to build
them from source and mount into the container.
```bash
git clone https://github.com/apache/iggy.git && cd iggy
cargo build --libs --release
# this will produce, among many: target/release/libiggy_connector_iceberg_sink.so
```
mount them via docker compose:
```
services:
iggy-connect:
image: apache/iggy-connect
volumes:
- ./target/release:/plugins:ro
- ./my-config:/config:ro
environment:
IGGY_CONNECTORS_CONFIG_PATH: /config/config.toml
```
and point your connector config at the mount path (extension is appended automatically):
path = "/plugins/libiggy_connector_iceberg_sink"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Connector Runtime: IoError(Unsupported) on Linux Metadata and Recursive Plugin Loading Loop [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on issue #2712:
URL: https://github.com/apache/iggy/issues/2712#issuecomment-3877682334
also, please use newest versions of everything (origin/master) 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Connector Runtime: IoError(Unsupported) on Linux Metadata and Recursive Plugin Loading Loop [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on issue #2712:
URL: https://github.com/apache/iggy/issues/2712#issuecomment-3877302834
just to make sure we're on the same page - the connectors runtime (binary `iggy-connectors`) is a host process that loads connector plugins (sink/source) as shared libraries (`.so` on Linux, `.dylib` on mac) at startup via dlopen. theplugin `.so` files are separate build artifacts from the runtime binary. the path field in your connector config toml tells the runtime where to find them.
if you need some reasoning about why we choose dynamic loading, I gladly invite you to discord so we can talk 😄
(main points are independent release cycles,language-agnostic interface - you can write plugin with any language that allows you to call C code)
a few questions about your setup:
- how are you running the connectors runtime? built from source, docker image, prebuilt binary?
- do you actually have the plugin `.so` files built and on disk? the runtime loads connector plugins as shared libraries via dlopen at startup - they're separate from the runtime binary itself.
- what does the path field in your connector config toml look like? absolute or relative?
- can you check if the .so file actually exists at that location (or next to the runtime binary)?
about point 1:
good catch - fixed on branch `iceberg-fix`. we now fall back to modified() when created() isn't supported
about point 2:
I improved the plugin path resolution.relative paths now search next to the runtime binary, cwd, /usr/lib, /usr/lib64, and /lib. if nothing is found you'll get an error listing all the paths we tried instead of a cryptic dlopen message.
See #2713
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Connector Runtime: IoError(Unsupported) on Linux Metadata and Recursive Plugin Loading Loop [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio closed issue #2712: [Bug] Connector Runtime: IoError(Unsupported) on Linux Metadata and Recursive Plugin Loading Loop
URL: https://github.com/apache/iggy/issues/2712
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org