[DISCUSS] Proposal to Start Development of a New Java Library in Apache Incubator

[Imre Tabur <im...@mail.ee.INVALID>]

Dear Apache Incubator Community,

I would like to start a discussion about an idea to start the 
development of a new Java library, designed to simplify
the implementation and setup of custom communication protocols, 
particularly for networked environments and device
interactions (e.g., serial ports, proprietary network protocols).

A library for building and setting up I/O (read/write, send/receive) 
protocols. Custom, non-standard,
proprietary, domain-, and vendor-specific protocol implementations, 
while remaining loosely coupled from the data
structures (message data structures) being transferred.

## Background: Why is this Needed?

In many projects, developers must work with proprietary or 
vendor-specific communication I/O protocols that are not
based on widely supported implementations like HTTP (REST, GraphQL, 
etc.) for which libraries exist. And the protocol
needs to be implemented by developers themselves. Shortly: no libraries 
to take.

### Use cases

* Integrating with (legacy) serial port devices in industrial systems.
* Implementing vendor-specific communication for IoT gateways or edge 
devices.
* Interacting with specialized network or hardware devices in 
telecommunications or data centers.
* Developing custom communication protocols for specific use cases, 
between devices and systems.

Such protocols often require developers to implement custom solutions 
from scratch, following detailed technical
documentation or specifications provided by the (protocol) vendor.

The problem is that doing it yourself is labor-intensive, requires 
precision, and is tedious work. The larger the
specification, the greater the struggle with implementation. It is also 
quite difficult to recall and follow what was
done later on. Additionally, there is little reusable code, and 
identifying parts of the code that can be reused is also
difficult.

This project aims to address these issues by providing a reusable 
library that simplifies working with such
vendor-specific protocols, reducing development time and ensuring 
consistency across implementations.

# Proposed Solution: A Graph-Based Approach to Protocols

The proposed library simplifies the development of custom client-server 
(sender-receiver and vice versa) communication
protocols by representing them as execution graphs.

In this model:

* Nodes in the graph represent protocol steps - reading or writing data 
(receive or send).
* Edges define the flow of execution between steps, governed by 
configurable rules and conditions.

The library provides functionality to build and execute graphs, handle 
basic I/O operations (and potentially more),
bind data structures to steps (or with the whole defined graph) to carry 
over protocol, handle events, and manage a
callback system.

**I assume that a graph, even when implemented in (fluent) code, is more 
readable (repairable/improvable) than a fully
custom implementation.**

By abstracting protocol logic into an execution graph, the library 
minimizes the complexity of handling vendor-specific
communication requirements and promotes easier maintenance.

The MVP implementation of the library will focus exclusively on Java to 
validate the concept and gather community
feedback before considering broader adoption or porting to other 
languages like C++ or other.

# Summary

To summarize, this project aims to simplify the implementation of 
non-standard protocols by providing a graph-based
framework that can handle basic I/O operations on its own through a 
defined graph. Most of the basic I/O functions are
handled within the library, compared to a custom-made solution where 
everything is cross-referenced and tightly coupled.

The initial focus will be on Java, with the potential for future 
expansion based on community input.

The initial base structure, skeleton, and proof of concept (PoC) will be 
independently developed by me and my team as a
preliminary effort in our own repository, serving as an example for 
further development and eventual transition into the
Apache Incubator.

# Feedback Request

The idea is still quite abstract, but I would greatly appreciate your 
feedback:

- Do you see value in such a project?
- Does this project align with ASF's ecosystem and goals?
- Is it potential ASF software?
- Are there existing projects or initiatives that we should consider or 
learn from?

Looking ahead:

- Would anyone be interested in collaborating, contributing or providing 
feedback during development?
- Would anyone be interested in mentoring?

If this aligns with your interests or expertise, please share your 
thoughts, questions, or suggestions.

Thank you!

Best regards,
Imre Tabur


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [DISCUSS] Proposal to Start Development of a New Java Library in Apache Incubator

[PJ Fanning <fa...@apache.org>]

Thanks for the detailed proposal.

Generally, the Apache Incubator takes existing projects with a pre-existing community. We emphasise community over code so it may be feasible to start a new project if you have a community.

Do you have a list of people who would be willing to collaborate on this? The existing Incubator contributor pool, generally, doesn't act as a pool of people who jump onto new projects. There is the concept of a mentor who helps new projects to get up and running - more from an admin and setup perspective.

On 2025/03/14 06:26:50 Imre Tabur wrote:
> Dear Apache Incubator Community,
> 
> I would like to start a discussion about an idea to start the 
> development of a new Java library, designed to simplify
> the implementation and setup of custom communication protocols, 
> particularly for networked environments and device
> interactions (e.g., serial ports, proprietary network protocols).
> 
> A library for building and setting up I/O (read/write, send/receive) 
> protocols. Custom, non-standard,
> proprietary, domain-, and vendor-specific protocol implementations, 
> while remaining loosely coupled from the data
> structures (message data structures) being transferred.
> 
> ## Background: Why is this Needed?
> 
> In many projects, developers must work with proprietary or 
> vendor-specific communication I/O protocols that are not
> based on widely supported implementations like HTTP (REST, GraphQL, 
> etc.) for which libraries exist. And the protocol
> needs to be implemented by developers themselves. Shortly: no libraries 
> to take.
> 
> ### Use cases
> 
> * Integrating with (legacy) serial port devices in industrial systems.
> * Implementing vendor-specific communication for IoT gateways or edge 
> devices.
> * Interacting with specialized network or hardware devices in 
> telecommunications or data centers.
> * Developing custom communication protocols for specific use cases, 
> between devices and systems.
> 
> Such protocols often require developers to implement custom solutions 
> from scratch, following detailed technical
> documentation or specifications provided by the (protocol) vendor.
> 
> The problem is that doing it yourself is labor-intensive, requires 
> precision, and is tedious work. The larger the
> specification, the greater the struggle with implementation. It is also 
> quite difficult to recall and follow what was
> done later on. Additionally, there is little reusable code, and 
> identifying parts of the code that can be reused is also
> difficult.
> 
> This project aims to address these issues by providing a reusable 
> library that simplifies working with such
> vendor-specific protocols, reducing development time and ensuring 
> consistency across implementations.
> 
> # Proposed Solution: A Graph-Based Approach to Protocols
> 
> The proposed library simplifies the development of custom client-server 
> (sender-receiver and vice versa) communication
> protocols by representing them as execution graphs.
> 
> In this model:
> 
> * Nodes in the graph represent protocol steps - reading or writing data 
> (receive or send).
> * Edges define the flow of execution between steps, governed by 
> configurable rules and conditions.
> 
> The library provides functionality to build and execute graphs, handle 
> basic I/O operations (and potentially more),
> bind data structures to steps (or with the whole defined graph) to carry 
> over protocol, handle events, and manage a
> callback system.
> 
> **I assume that a graph, even when implemented in (fluent) code, is more 
> readable (repairable/improvable) than a fully
> custom implementation.**
> 
> By abstracting protocol logic into an execution graph, the library 
> minimizes the complexity of handling vendor-specific
> communication requirements and promotes easier maintenance.
> 
> The MVP implementation of the library will focus exclusively on Java to 
> validate the concept and gather community
> feedback before considering broader adoption or porting to other 
> languages like C++ or other.
> 
> # Summary
> 
> To summarize, this project aims to simplify the implementation of 
> non-standard protocols by providing a graph-based
> framework that can handle basic I/O operations on its own through a 
> defined graph. Most of the basic I/O functions are
> handled within the library, compared to a custom-made solution where 
> everything is cross-referenced and tightly coupled.
> 
> The initial focus will be on Java, with the potential for future 
> expansion based on community input.
> 
> The initial base structure, skeleton, and proof of concept (PoC) will be 
> independently developed by me and my team as a
> preliminary effort in our own repository, serving as an example for 
> further development and eventual transition into the
> Apache Incubator.
> 
> # Feedback Request
> 
> The idea is still quite abstract, but I would greatly appreciate your 
> feedback:
> 
> - Do you see value in such a project?
> - Does this project align with ASF's ecosystem and goals?
> - Is it potential ASF software?
> - Are there existing projects or initiatives that we should consider or 
> learn from?
> 
> Looking ahead:
> 
> - Would anyone be interested in collaborating, contributing or providing 
> feedback during development?
> - Would anyone be interested in mentoring?
> 
> If this aligns with your interests or expertise, please share your 
> thoughts, questions, or suggestions.
> 
> Thank you!
> 
> Best regards,
> Imre Tabur
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org