Proposal to Connector Architecture in Synapse Go

[Thisara Weerakoon <th...@gmail.com>]

Hi all,

As we continue developing the Go-based implementation of Apache Synapse,
establishing a robust mechanism for extending its capabilities through
connectors is crucial.

Connectors are fundamental to Synapse, allowing easy connection to diverse
systems (like SaaS, databases), abstracting complexity, and making the
platform more adaptable and user-friendly for building integrations.

Regarding connector implementation in Go Synapse, many can likely be built
using existing mediation building blocks via configuration, avoiding the
need for custom Go code. However, when integration requirements become too
complex or specialized for these standard mediators, implementing the
connector logic with custom Go code will be necessary. It is specifically
this requirement—to dynamically load and integrate these custom Go
code-based connectors at runtime—that necessitates exploring suitable
extension mechanisms within the Go ecosystem.

Therefore, to achieve similar dynamic extensibility in our Go
implementation, I've looked into suitable patterns within the Go ecosystem.
I propose that we consider leveraging the standard Go plugin package
<https://pkg.go.dev/plugin> provided in Go's standard library.

Conceptually, this approach works as follows:

   - Individual connectors would be developed as separate Go packages.
   - These packages would be compiled specifically into shared object (.so)
   files using the *go build -buildmode=plugin* command.
   - The main Go Synapse engine would then be able to discover and
   dynamically load these .so files at runtime using functions provided by
   the plugin package (like *plugin.Open*).
   - Once loaded, the engine could look up specific, pre-defined functions
   or variables exported by the connector's code.
   - Interaction between the engine and the connector logic would occur
   through standard Go interfaces, allowing the engine to invoke connector
   operations (like initialization or execution) with the message context.

To help visualize this, I have prepared a conceptual design diagram
outlining this architecture (please see attached). Furthermore, to
demonstrate the basic mechanics, I have implemented a simple
Proof-of-Concept code elaborating this approach using the plugin package.
You can find the code here:
https://github.com/ThisaraWeerakoon/go-plugin-example

I believe leveraging the standard plugin package
<https://pkg.go.dev/plugin> presents
a viable path forward for implementing dynamic connectors in Go Synapse. I
am keen to hear your thoughts, feedback, and any potential considerations
or concerns regarding this proposed architecture. Please let me know what
you think. I look forward to starting a discussion on this topic to
determine the best direction for our connector framework.

-- 
Thisara Rathnayaka
Undergraduate Student | Computer Science & Engineering
University of Moratuwa
LinkedIn <http://www.linkedin.com/in/weerakoon-thisara-rathnayaka-881826177>
| GitHub <https://github.com/ThisaraWeerakoon> | Blog
<ht...@thisara.weerakoon2001>