You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "Tyooughtul (via GitHub)" <gi...@apache.org> on 2026/04/20 01:00:57 UTC
[PR] Feat/1846 avro support [iggy]
Tyooughtul opened a new pull request, #3141:
URL: https://github.com/apache/iggy/pull/3141
## Which issue does this PR close?
Closes #1846
## Rationale
Issue #1846 requested support for Apache Avro as a first-class payload format in connectors, including dedicated StreamEncoder / StreamDecoder implementations and cross-format transformations. This PR delivers that support with schema-driven encoding/decoding and runtime integration.
## What changed?
## What changed?
Connectors only supported JSON, Raw, Text, Proto and FlatBuffer payloads. Avro data had to be handled as opaque raw bytes without schema validation or format conversion.
So, I added full Avro support across the connector stack, SDK-level AvroStreamEncoder/Decoder with inline/file-based schema loading, a new AvroConvert transform for cross-format conversion, runtime config fields for producers/consumers, and HTTP sink base64 encoding for Avro payloads.
## Local Execution
- Passed / not passed
Passed
- Pre-commit hooks ran / not ran
Ran
## AI Usage
I used codex to check the code I wrote, and it helped me discover some bugs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(connectors): implement Avro payload support with separate encoder, decoder and transform crates [iggy]
Posted by "spetz (via GitHub)" <gi...@apache.org>.
spetz merged PR #3141:
URL: https://github.com/apache/iggy/pull/3141
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feat/1846 avro support [iggy]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #3141:
URL: https://github.com/apache/iggy/pull/3141#issuecomment-4277267646
## [Codecov](https://app.codecov.io/gh/apache/iggy/pull/3141?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 72.06%. Comparing base ([`2d6562b`](https://app.codecov.io/gh/apache/iggy/commit/2d6562b7fd242a36bdf57a231a836bc6b80d6107?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`e426d71`](https://app.codecov.io/gh/apache/iggy/commit/e426d71529985cc4912d5ddc7a61b3ae9cee6549?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #3141 +/- ##
============================================
- Coverage 73.17% 72.06% -1.11%
Complexity 943 943
============================================
Files 1123 1060 -63
Lines 97892 93078 -4814
Branches 75065 74972 -93
============================================
- Hits 71632 67080 -4552
+ Misses 23671 23409 -262
Partials 2589 2589
```
| [Components](https://app.codecov.io/gh/apache/iggy/pull/3141/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [Rust Core](https://app.codecov.io/gh/apache/iggy/pull/3141/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.05% <ø> (-0.01%)` | :arrow_down: |
| [Java SDK](https://app.codecov.io/gh/apache/iggy/pull/3141/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.30% <ø> (ø)` | |
| [C# SDK](https://app.codecov.io/gh/apache/iggy/pull/3141/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `69.40% <ø> (ø)` | |
| [Python SDK](https://app.codecov.io/gh/apache/iggy/pull/3141/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.43% <ø> (ø)` | |
| [Node SDK](https://app.codecov.io/gh/apache/iggy/pull/3141/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `87.29% <ø> (-4.01%)` | :arrow_down: |
| [Go SDK](https://app.codecov.io/gh/apache/iggy/pull/3141/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `39.41% <ø> (ø)` | |
| [Files with missing lines](https://app.codecov.io/gh/apache/iggy/pull/3141?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [core/connectors/runtime/src/configs/connectors.rs](https://app.codecov.io/gh/apache/iggy/pull/3141?src=pr&el=tree&filepath=core%2Fconnectors%2Fruntime%2Fsrc%2Fconfigs%2Fconnectors.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jb25uZWN0b3JzL3J1bnRpbWUvc3JjL2NvbmZpZ3MvY29ubmVjdG9ycy5ycw==) | `28.18% <ø> (ø)` | |
| [core/connectors/runtime/src/sink.rs](https://app.codecov.io/gh/apache/iggy/pull/3141?src=pr&el=tree&filepath=core%2Fconnectors%2Fruntime%2Fsrc%2Fsink.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jb25uZWN0b3JzL3J1bnRpbWUvc3JjL3NpbmsucnM=) | `74.14% <ø> (-0.38%)` | :arrow_down: |
| [core/connectors/runtime/src/source.rs](https://app.codecov.io/gh/apache/iggy/pull/3141?src=pr&el=tree&filepath=core%2Fconnectors%2Fruntime%2Fsrc%2Fsource.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jb25uZWN0b3JzL3J1bnRpbWUvc3JjL3NvdXJjZS5ycw==) | `69.41% <ø> (-0.08%)` | :arrow_down: |
| [core/connectors/sdk/src/encoders/flatbuffer.rs](https://app.codecov.io/gh/apache/iggy/pull/3141?src=pr&el=tree&filepath=core%2Fconnectors%2Fsdk%2Fsrc%2Fencoders%2Fflatbuffer.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jb25uZWN0b3JzL3Nkay9zcmMvZW5jb2RlcnMvZmxhdGJ1ZmZlci5ycw==) | `80.99% <ø> (ø)` | |
| [core/connectors/sdk/src/encoders/proto.rs](https://app.codecov.io/gh/apache/iggy/pull/3141?src=pr&el=tree&filepath=core%2Fconnectors%2Fsdk%2Fsrc%2Fencoders%2Fproto.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jb25uZWN0b3JzL3Nkay9zcmMvZW5jb2RlcnMvcHJvdG8ucnM=) | `71.62% <ø> (ø)` | |
| [core/connectors/sdk/src/lib.rs](https://app.codecov.io/gh/apache/iggy/pull/3141?src=pr&el=tree&filepath=core%2Fconnectors%2Fsdk%2Fsrc%2Flib.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jb25uZWN0b3JzL3Nkay9zcmMvbGliLnJz) | `60.24% <ø> (ø)` | |
| [core/connectors/sdk/src/transforms/mod.rs](https://app.codecov.io/gh/apache/iggy/pull/3141?src=pr&el=tree&filepath=core%2Fconnectors%2Fsdk%2Fsrc%2Ftransforms%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jb25uZWN0b3JzL3Nkay9zcmMvdHJhbnNmb3Jtcy9tb2QucnM=) | `33.33% <ø> (ø)` | |
| [...ore/connectors/sdk/src/transforms/proto\_convert.rs](https://app.codecov.io/gh/apache/iggy/pull/3141?src=pr&el=tree&filepath=core%2Fconnectors%2Fsdk%2Fsrc%2Ftransforms%2Fproto_convert.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jb25uZWN0b3JzL3Nkay9zcmMvdHJhbnNmb3Jtcy9wcm90b19jb252ZXJ0LnJz) | `73.02% <ø> (ø)` | |
| [core/connectors/sinks/http\_sink/src/lib.rs](https://app.codecov.io/gh/apache/iggy/pull/3141?src=pr&el=tree&filepath=core%2Fconnectors%2Fsinks%2Fhttp_sink%2Fsrc%2Flib.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jb25uZWN0b3JzL3NpbmtzL2h0dHBfc2luay9zcmMvbGliLnJz) | `86.28% <ø> (ø)` | |
... and [81 files with indirect coverage changes](https://app.codecov.io/gh/apache/iggy/pull/3141/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org