[jira] [Created] (CAMEL-21413) Create a TransactionLog transaction log to provide detailed tracking and performance metrics on end-to-end transactions

["Raymond (Jira)" <ji...@apache.org>]

Raymond created CAMEL-21413:
-------------------------------

             Summary: Create a TransactionLog transaction log to provide detailed tracking and performance metrics on end-to-end transactions 
                 Key: CAMEL-21413
                 URL: https://issues.apache.org/jira/browse/CAMEL-21413
             Project: Camel
          Issue Type: Wish
          Components: came-core
            Reporter: Raymond
         Attachments: TransactionStore_TrackAndTrace.jpg

*Wish*

The wish is for a transaction log store in Camel to provide detailed tracking and performance metrics on end-to-end transactions across multiple exchanges/routes, storing metadata, exchange data, and performance metrics in a configurable, persistent registry to better monitor business processes.


*Background*

A lot of data in tracing and observability is on the exchange level. How many exchanges are completed or failed, for example. In many cases, what I really want to know if a transaction is successful or failed.

With a transaction I mean that the complete lifecycle of a message on Camel, this 
may contain many routes and exchanges. For example, when Seda or ActiveMQ is used 
a new exchange is created, but it's still the same transaction.
 
To have metrics and more insights on transactions, I would like to have a transaction log (through a separate registry?). This is a separate store with information about the transaction. 


*Implementation ideas*


Below are few ideas to implement this wish:

_1. Initialize the store_

Setup a transaction log/store (either in memory or external store) through a registry. This is to enable it and to configure the store.


_2. Transaction item_

A transaction item is a definition of a specific transaction. It may contain the following items:
 * {*}Name{*}: The name to indicate the transaction. Like "order" or "invoice". This is so to say the overarching name of the integration/business process.
 * {*}StartingPoint{*}: A routeId or groupId where the transaction starts (triggers a new transaction).
 * {*}CorrelationID(s){*}: One or more correlationId (headers) that can be set just like a header/exchangePropery/variable, this contains for example the orderID that is set on the first step of the transaction.


_3. Tracking transactions_

A transaction is probably best tracked by something like a breadcrumbId. Preferably a transactionId. At least something that cannot be edited by users (I sometimes see people change the breadcrumbId, to separate transactions into multiple segments).

By default, only the start and end of a transaction is tracked. To get more fine-grained control, one could also track intermediate points like a route, a step, or a node.

The transaction is tracked during processing, and then finished on completion. The whole counts as one "UnitOfWork".

_4. The transaction store_

The transaction store contains all the tracked items. Maybe there could be divided into separate segments like:
 # metadata

    - name (name of the transaction)
    - description (description of type of transaction)
    - transactionId
    - transactionHistory (similar to messageHistory)
    - correlationId(s)
    - timestamps
    - exceptions
 # exchange data

    - headers, properties, variables
    - bodyType
    - body
    etc


 # performance metrics

    - load (CPU), memory (RAM), threads
    - number of transactions (completed, pending, failed)
    - throughput
    - response time
    - total process time of an transaction
    - average
    - etc

It should be possible to configure which of these items the user what want to store. By default, only metadata.

_5. Transaction API's_

There need to be an API to configure and operate the transaction store. (including JMX), similar to backlogtracer.

*Production*
Some of the solutions I saw are only available during development or test, but the need is especially big for production environment. A solution needs to keep in mind that this would run for billion of messages. The flexibility to turn it easily on and off.

For example, by default, the body of a message doesn't need to be stored. Or only the first 32kb need to be stored. But for specific case users, one to enable this on the fly for a specific transaction.

Transactions are probably needed persistent, so either need a messagestore (disk) similar to brokers or an external database. I'm not sure what the best stores are for this kind of data. A normal SQL database like Postgres, a search database like Elastic, a metrics database like Prometheus or key/value store like Redis?



*End users*

The idea is that end users have a good insight in what is going on in the integration layer. So that they can question like the following:
 * How many order transactions are processed?
 * Give me a specific order by orderId that custom says he didn't receive?
 * How many failed messages are there? Which ones did go wrong, where did it go wrong, and what is the exact error?
 * Can we replay the failed order?
 * Where is the bottleneck on the system

*Conclusion*

This is just an idea/wish, but something I see coming back from the start of my career as integration specialist (around 2007), until today. 

Most consultancy firms or end customers have built something themselves, I see with names like "Track & Trace", "MessageStore", "ESB Collector", "Transactions", "Control center".

I also built a solution myself, and are happy to share this solution, or others I saw. A more generic solution to monitor (track & trace) transactions of business processes would be very welcome.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)