[DISCUSS] New Database Storage Layer for Artemis

[Clebert Suconic <cl...@gmail.com>]

I would like to share some exciting work I have been doing.
It's a new database layer for Apache Artemis.
It is currently sitting on my fork, branch new-storage:
- https://github.com/clebertsuconic/artemis/tree/new-storage

It is a total redesign of how we interact with JDBC. The current
version of JDBC is a layer mapping our file store system into tables.
It served a purpose for certain users, but it was never a first-class
storage option for the broker.

This storage layer is actually surprisingly good. I'm getting
performance numbers better than I would get with our journal system in
certain cases. (At least when I use PostgreSQL as the RDBMS).
I am playing with a shared pool of connections. Each connection is
associated with a worker, that collects multiple operations, batching
into the database in a very optimized way. I am using the async
completion model from the Journal system into this database layer, and
the results are surprisingly good.

I still need to implement large message support and XA transactions,
but things are starting to get into shape.

And I'm looking for volunteers to try it out, and collaborate.

To get started, you can download my fork and build it:

git clone https://github.com/clebertsuconic/artemis.git
cd artemis
git checkout new-storage
mvn install -DskipTests=true -T1C

Start PostgreSQL using the scripts:

cd artemis/tests/db-tests/scripts
./start-postgres.sh

There's a server pre-created under the test environment you can use to
play with it:
(It is important you use new-postgres, as postgres would be the
old/current model)

cd artemis/tests/db-tests/target/new-postgres/bin
./artemis run

On another screen or terminal you can use the CLI and play with
producer / consumer, or even with your clients.

It currently supports all the protocols, 7 database types. I will work
on large message support next week, and look to implement XA. But I
would love collaborations already.

I can dig in more on the design as we progress on this thread.
I've documented the architecture and patterns in the branch -- see
AGENTS.md for an overview of the design.

I am copying both Dev and Users as I'm looking to collaborate on this
with both users who would be willing to test it, and devs who would
share testing and feedback.


--
Clebert Suconic

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@artemis.apache.org
For additional commands, e-mail: dev-help@artemis.apache.org