Re: [DISCUSSION] Performance issues with data-index persistence addon

[Alex Porcelli <al...@porcelli.me>]

Any update or new findings on this topic?

On Tue, Nov 28, 2023 at 8:38 AM Francisco Javier Tirado Sarti
<ft...@redhat.com> wrote:
>
> Hi Alex,
> After considering different options to improve performance, we feel that it
> is time to "partially" move away from the current Map style interface (
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java)
> which was shared with Trusty, to one more suitable for usage with a
> relational DB like postgresql (but still compatible with big table dbs).
> The idea will be to replace generic Storage interface by four specific
> interfaces (which will inherit from a common one that keeps the query part
> at is it. with get and query methods), that will include the required
> modification operations for the four DataIndex "domains": processinstance,
> usertask, processdefinitions and jobs. Those interfaces will define methods
> like addNode, addVariable, updateTask, addAttachment..... that will allow
> the persistent layer implementation  to just update the needed info in the
> DB  (for example, for addNode in Postgres, just insert a row into nodes
> table, for addNode in Mongo, basically the same atomic upsert operation
> that is currently done). Therefore, we increase performance for Postgres
> and keep the current one for Mongo. The current DB schemas won't be
> touched.
> Since the code change is large, I do not think I'll be able to have the PR
> ready till next week.
> But before starting, please let me know if that approach is fine for you.
> Best regards.
>
>
>
>
> On Fri, Nov 24, 2023 at 6:55 PM Alex Porcelli <al...@porcelli.me> wrote:
>
> > Thank you Francisco to getting deeper on this…
> >
> > Looking forward to see the results of your suggested improvements.
> >
> >
> > On Fri, Nov 24, 2023 at 9:40 AM Francisco Javier Tirado Sarti <
> > ftirados@redhat.com> wrote:
> >
> > > I forgot to attach the queries
> > >
> > > On Fri, Nov 24, 2023 at 3:04 PM Francisco Javier Tirado Sarti <
> > > ftirados@redhat.com> wrote:
> > >
> > >> Hi,
> > >> A brief update on this topic.
> > >> After doing a simple test with example
> > >>
> > https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > ,
> > >> the number of updates over Nodes table is n*n, so we manage to obtain a
> > >> perfect quadratic performance degradation. The problem is worse in the
> > case
> > >> of Serverless Workflow than in BPMN because we the number of nodes is
> > >> greater than the number of states. In that example N is 16, but for a
> > more
> > >> complex workflow it would be certainly large.
> > >> I think that this is more related to how we are handling JPA in the
> > code,
> > >> in particular the mapping from model to entity (basically JPA is blind
> > and
> > >> has to update all nodes for every write because it believes the node has
> > >> been updated, although it is not) than an issue in the table definition.
> > >> In fact, when using JPA, separating the server model from the JPA
> > entity is
> > >> not a good idea, especially if the entity contains collections. I will
> > try
> > >> to change that without breaking anything.
> > >>
> > >> On Wed, Nov 22, 2023 at 12:10 PM Enrique Gonzalez Martinez <
> > >> egonzalez@apache.org> wrote:
> > >>
> > >>> After the events split you now will need to create a node instance
> > >>> model instance of making independent from the process instance.
> > >>> That should do the trick.
> > >>>
> > >>> Regarding deleting/inserting it was fixed at some point.
> > >>>
> > >>> El mar, 21 nov 2023 a las 20:22, Francisco Javier Tirado Sarti
> > >>> (<ft...@redhat.com>) escribió:
> > >>> >
> > >>> > Hi Martin,
> > >>> > I have a task to review performance of
> > >>> >
> > >>> > ProcessInstanceNodeDataEventMerger
> > >>> > My idea is to reduce the number of delete inserts when processing
> > >>> events
> > >>> > and try to do it incremental.
> > >>> > That should improve performance.
> > >>> > PS:
> > >>> > I was planning to send an e-mail tomorrow announcing that in case you
> > >>> were
> > >>> > already working on a fix for that. I assume you are not and I would
> > be
> > >>> > sending a PR soon.
> > >>> >
> > >>> > On Tue, Nov 21, 2023 at 6:09 PM Martin Weiler
> > <mweiler@ibm.com.invalid
> > >>> >
> > >>> > wrote:
> > >>> >
> > >>> > > I looked into the new examples using data-index persistence addon -
> > >>> Neus'
> > >>> > > PR#1813 [1] for serverless and Pere's branch [2] for workflow
> > (great
> > >>> job
> > >>> > > both!) - and they work without issues using single requests.
> > >>> However, under
> > >>> > > some load (I used 'ab' for testing with a light concurrency of 10
> > >>> parallel
> > >>> > > requests) I ran into the following problems:
> > >>> > >
> > >>> > > (1) Large number of insert/delete calls (eg. for tables such as
> > >>> nodes,
> > >>> > > definitions, etc)
> > >>> > >
> > >>> > > (2) Hibernate OptimisticLockExceptions / StaleStateExceptions
> > >>> > >
> > >>> > > (3) DB deadlocks
> > >>> > >
> > >>> > > (4) Error responses, slow response times
> > >>> > >
> > >>> > > The reason I am reaching out with this topic here is to find out if
> > >>> we are
> > >>> > > aware of this issue, and if someone is already looking into or
> > being
> > >>> > > assigned to it?
> > >>> > >
> > >>> > > Thanks,
> > >>> > > Martin
> > >>> > >
> > >>> > > [1]
> > >>> https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > >>> > > [2]
> > >>> > >
> > >>>
> > https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > >>> > >
> > >>> > >
> > ---------------------------------------------------------------------
> > >>> > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > >>> > > For additional commands, e-mail: dev-help@kie.apache.org
> > >>> > >
> > >>> > >
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > >>> For additional commands, e-mail: dev-help@kie.apache.org
> > >>>
> > >>>
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > For additional commands, e-mail: dev-help@kie.apache.org
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
For additional commands, e-mail: dev-help@kie.apache.org

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Francisco Javier Tirado Sarti <ft...@redhat.com>]

Yes, I understood the idea was to use k8s cronjob (
https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/) , but
we should not allow users to setup DB deletion on their own, therefore we
need to provide the script  cronjob manifest will invoke to delete the db
rows (and one of the parameters of the script has to be the duration
interval). I got your point, this way we avoid adding a background thread
to our microservice dedicated to the purge (and base on a config value).
About bulk, note that both operations I described in my latest e-mail ( a
list of process instances to be deleted or a calculated list of process
instances to be deleted based on completion date) are bulk. My point was
that we need to carefully decide which  BULK API this REST interface
provides.

On Thu, Feb 22, 2024 at 12:18 PM Alex Porcelli <po...@apache.org> wrote:

> Francisco,
>
> We wont use cron ourselves, we’ll recommend to users if they want a regular
> cleanup. We don’t need to own this, Kubernetes has native cron and OSs also
> have their native cron…. So we don’t need replicate such functionality.
>
> I think the cleanup we are talking about here is a bulk operation,
> individual instances deletion should be handled differently.
>
> On Thu, Feb 22, 2024 at 6:03 AM Francisco Javier Tirado Sarti <
> ftirados@redhat.com> wrote:
>
> > Ok, so we have two flavours, clean up on demand through REST user call
> > (that's fine, the only issue is the scope of it. For example, should we
> > allow a bulk delete api where you specify the process instances, if the
> > user knows them somehow, that user wants to delete or just, as we are
> > proposing, delete all instances that have been there for a while?) and
> cron
> > process (that is a replacement of the properties approach I recommend).
> > In my opinion, if we have the explicit REST API, we should not use the
> > cron.
> >
> > On Thu, Feb 22, 2024 at 6:32 AM Enrique Gonzalez Martinez <
> > egonzalez@apache.org> wrote:
> >
> > > +1 alex
> > >
> > > El jue, 22 feb 2024, 0:14, Alex Porcelli <al...@porcelli.me> escribió:
> > >
> > > > I think I need to clarify what I mean by REST...
> > > >
> > > > The REST endpoint I have in mind would execute the necessary cleaning
> > > > based on an input parameter (ie. instanced closed for more than X
> days
> > > > or something like that).
> > > >
> > > > The use of cron is that an external cron could call the cleaning REST
> > > > endpoint from time to time (once a month, on every sunday night,
> etc).
> > > >
> > > >
> > > > On Wed, Feb 21, 2024 at 10:50 AM Francisco Javier Tirado Sarti
> > > > <ft...@redhat.com> wrote:
> > > > >
> > > > > Hi Alex,
> > > > > I'm wondering how cron and rest can live together.
> > > > > With cron the responsibility to do the clean up relies on a process
> > > > > external to the microservice accessing the db to be purged, while
> > where
> > > > > using REST API the process performing the clean up is typically
> (but
> > > not
> > > > > necessarily) the same microservice that accesses the DB. In that
> > sense,
> > > > > REST API is similar to using properties (in both approaches the
> > > > responsible
> > > > > to clean up the data is the microservice), but they are different
> > > because
> > > > > REST is more sophisticated (but compatible, implementation wise we
> > can
> > > > use
> > > > > a ConfigSource, that is updated by the REST and read, as a
> property,
> > by
> > > > the
> > > > > cleanup routine). So, probably I'm missing something but I see the
> > cron
> > > > > approach not reconcilable with the REST approach.
> > > > > Anyway just a detail, I agree the purge approach should be similar
> > for
> > > > all
> > > > > microservices
> > > > >
> > > > > On Wed, Feb 21, 2024 at 4:37 PM Alex Porcelli <porcelli@apache.org
> >
> > > > wrote:
> > > > >
> > > > > > +1 for provide consistent behavior across the board
> > > > > >
> > > > > > my 2c:
> > > > > >
> > > > > > +1 for REST interface (preferred for external general management
> > > > > > perspective)
> > > > > > -1 for properties to auto clean; this can be achieved with cron
> and
> > > > rest
> > > > > > call
> > > > > >
> > > > > > Now about GraphQL: the platform today is heavily invested in such
> > > > > > interface, so it might make sense. BUT I’d consider it in a
> second
> > > > moment.
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 21, 2024 at 6:52 AM Francisco Javier Tirado Sarti <
> > > > > > ftirados@redhat.com> wrote:
> > > > > >
> > > > > > > Hi Enrique,
> > > > > > > If we configure such policies through properties, it won't be
> > > enough
> > > > to
> > > > > > > define a naming convention and rely on the target platform
> > > > configuration
> > > > > > > capabilities (SpringBoot or Quarkus)?.
> > > > > > >
> > > > > > > On Wed, Feb 21, 2024 at 12:36 PM Enrique Gonzalez Martinez <
> > > > > > > egonzalez@apache.org> wrote:
> > > > > > >
> > > > > > > > Hi Francisco
> > > > > > > >
> > > > > > > > The discussion we need to have before how to achieve certain
> > > > features,
> > > > > > is
> > > > > > > > the overall of the user experience. If you want to clean it
> up
> > > > that way
> > > > > > > it
> > > > > > > > is fine by me. I have nothing against setting policy related
> to
> > > > some
> > > > > > sort
> > > > > > > > of clean up for a process completed some time ago.
> > > > > > > >
> > > > > > > > As we discussed previously the data index is a snapshot of
> the
> > > last
> > > > > > state
> > > > > > > > of the process instance included completed, but nothing was
> > said
> > > > once
> > > > > > > > completed when we need to clean that up so any policy is
> > welcome.
> > > > How
> > > > > > to
> > > > > > > > achieve that policy is something completely different.
> > > > > > > >
> > > > > > > > The main problem is how every microservice is exposing that
> API
> > > to
> > > > the
> > > > > > > end
> > > > > > > > user or consumer (other system) and which system is going to
> be
> > > the
> > > > > > > façade
> > > > > > > > for complex deployments and operations. That is the
> discussion
> > I
> > > > was
> > > > > > > > mentioning before.
> > > > > > > >
> > > > > > > > What I want to avoid is to set different policies among
> > > components
> > > > and
> > > > > > we
> > > > > > > > should strive to be as much as possible to offer certain
> > > > capabilities
> > > > > > in
> > > > > > > > the same fashion, e.g: clean up mechanism.
> > > > > > > > In the same way I want to avoid making the current system
> more
> > > > complex
> > > > > > > than
> > > > > > > > they are. So far they are aligned offering one simple
> > > > responsibility
> > > > > > but
> > > > > > > I
> > > > > > > > would be against mixing for instance job service with data
> > index.
> > > > > > > >
> > > > > > > > El mié, 21 feb 2024 a las 12:13, Francisco Javier Tirado
> Sarti
> > (<
> > > > > > > > ftirados@redhat.com>) escribió:
> > > > > > > >
> > > > > > > > > Hi Enrique,
> > > > > > > > > In the case of data index I think the data to be purged is
> > > > finished
> > > > > > > > process
> > > > > > > > > instances (I do not think we should remove process instance
> > > that
> > > > has
> > > > > > > been
> > > > > > > > > alive for ages, even if it is very likely they are not
> going
> > to
> > > > be
> > > > > > ever
> > > > > > > > > completed)
> > > > > > > > > Once you delete those process instances, you also delete
> the
> > > > > > associated
> > > > > > > > > user tasks and jobs.
> > > > > > > > > Therefore the problem is relatively simple, to be able to
> > > > configure
> > > > > > how
> > > > > > > > > much time a completed process instance should remain in the
> > > data
> > > > > > index
> > > > > > > > > database. We can take a simple approach: a property with a
> > min
> > > > > > duration
> > > > > > > > > that cannot be changed once data index is started; a
> slightly
> > > > complex
> > > > > > > > one:
> > > > > > > > > the same property but watching it to react for changes; or
> > the
> > > > full
> > > > > > > > suite:
> > > > > > > > > an admin API to be able to change the policy at any moment.
> > > > > > > > > I think this discussion is worth having.
> > > > > > > > >
> > > > > > > > > On Wed, Feb 21, 2024 at 6:15 AM Enrique Gonzalez Martinez <
> > > > > > > > > egonzalez@apache.org> wrote:
> > > > > > > > >
> > > > > > > > > > Hi Martin, the main problem regarding the purge is
> because
> > it
> > > > is
> > > > > > > still
> > > > > > > > > > unclear the policy for what tech to use and the future
> > > > components.
> > > > > > > > > >
> > > > > > > > > > Recently we had a discussion about proposing graphql for
> > this
> > > > sort
> > > > > > of
> > > > > > > > > admin
> > > > > > > > > > tasks. So far for subsystems we have been using rest
> > > endpoints
> > > > > > (like
> > > > > > > > > update
> > > > > > > > > > timers or modify human task or change processes). There
> is
> > > one
> > > > > > > > exception
> > > > > > > > > > which is the gateway that is pure graphql and somehow
> uses
> > > > graphql
> > > > > > > for
> > > > > > > > > > everything making complex operations under the hood.
> > > > > > > > > >
> > > > > > > > > > This has somehow frozen the purge for data audit for a
> bit
> > > and
> > > > the
> > > > > > > > > proposal
> > > > > > > > > > was to use rest endpoints to do the clean up in the
> > component
> > > > and
> > > > > > > offer
> > > > > > > > > the
> > > > > > > > > > graphql counterpart in the gateway promoting it to a
> first
> > > > class
> > > > > > > > citizen
> > > > > > > > > > component instead of having it embedded in the data
> index.
> > > > > > > > > >
> > > > > > > > > > I would suggest to come up at least with a policy first
> > > > regarding
> > > > > > the
> > > > > > > > > > convention every component should address this.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > El mar, 20 feb 2024, 23:39, Martin Weiler
> > > > <mweiler@ibm.com.invalid
> > > > > > >
> > > > > > > > > > escribió:
> > > > > > > > > >
> > > > > > > > > > > IMO, it is good to have this discussion around data
> > sanity
> > > > now
> > > > > > > > instead
> > > > > > > > > of
> > > > > > > > > > > putting it off until later when data has already
> > > accumulated
> > > > in
> > > > > > > > > > production
> > > > > > > > > > > environments.
> > > > > > > > > > >
> > > > > > > > > > > Based on the input here, we are dealing with three
> types
> > of
> > > > data:
> > > > > > > > > > > 1. Runtime data - active instances only, engine cleans
> up
> > > the
> > > > > > data
> > > > > > > > > > > automatically at process instance end
> > > > > > > > > > > 2. Historic log data - data created by data-audit
> > intended
> > > > for
> > > > > > long
> > > > > > > > > term
> > > > > > > > > > > storage
> > > > > > > > > > > 3. Data-index data - somehow this data falls in between
> > the
> > > > two
> > > > > > > > > > > aforementioned categories, with the idea of the data
> > being
> > > > > > > "recent",
> > > > > > > > > but
> > > > > > > > > > > not restricted to active instances only
> > > > > > > > > > >
> > > > > > > > > > > We'd need purge strategies for both #2 and #3 (perhaps
> > > > different
> > > > > > > > ones,
> > > > > > > > > or
> > > > > > > > > > > with different config settings) in order to prevent
> > > unlimited
> > > > > > data
> > > > > > > > > > growth.
> > > > > > > > > > >
> > > > > > > > > > > ________________________________________
> > > > > > > > > > > From: Enrique Gonzalez Martinez <eg...@apache.org>
> > > > > > > > > > > Sent: Monday, February 19, 2024 7:11 AM
> > > > > > > > > > > To: dev@kie.apache.org
> > > > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues
> > > with
> > > > > > > > data-index
> > > > > > > > > > > persistence addon
> > > > > > > > > > >
> > > > > > > > > > > Hi Francisco,
> > > > > > > > > > > To give you more context about this.
> > > > > > > > > > >
> > > > > > > > > > > STP is a concept, a process with certain constraints:
> no
> > > > > > > persistence
> > > > > > > > > and
> > > > > > > > > > > returning the outcome in the call (sync execution with
> no
> > > > idle
> > > > > > > > states).
> > > > > > > > > > It
> > > > > > > > > > > was a requirement from a user in the past. One of the
> > > > > > requirements
> > > > > > > > was
> > > > > > > > > > > leaving no trail. In v7 was easy because you could
> > disable
> > > > the
> > > > > > > audit
> > > > > > > > in
> > > > > > > > > > > that case. Actually we have the same way to do what we
> > did
> > > > in v7
> > > > > > in
> > > > > > > > > here
> > > > > > > > > > as
> > > > > > > > > > > you can add/remove index just removing deps.
> > > > > > > > > > >
> > > > > > > > > > > We have the same outcome with different approaches and
> > STP
> > > is
> > > > > > > already
> > > > > > > > > > > delivered.
> > > > > > > > > > >
> > > > > > > > > > > El lun, 19 feb 2024 a las 14:46, Francisco Javier
> Tirado
> > > > Sarti (<
> > > > > > > > > > > ftirados@redhat.com>) escribió:
> > > > > > > > > > >
> > > > > > > > > > > > Regarding STP (which is not a concept that we have in
> > the
> > > > > > code. I
> > > > > > > > > mean
> > > > > > > > > > > STP
> > > > > > > > > > > > are processes as nonSTP are), I guess, as all
> > processes,
> > > > they
> > > > > > > were
> > > > > > > > > kept
> > > > > > > > > > > in
> > > > > > > > > > > > DataIndex once completed because users wanted (and
> > still
> > > > wants)
> > > > > > > to
> > > > > > > > > > check
> > > > > > > > > > > > the result once the call had been performed. If we
> want
> > > to
> > > > > > leave
> > > > > > > no
> > > > > > > > > > trace
> > > > > > > > > > > > of them in DataIndex for some reason, we will need to
> > > make
> > > > it a
> > > > > > > > > > > > Runtimes concept so DataIndex can handle them in a
> > > > different
> > > > > > way.
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Feb 19, 2024 at 2:27 PM Enrique Gonzalez
> > > Martinez <
> > > > > > > > > > > > egonzalez@apache.org> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Alex:
> > > > > > > > > > > > > Right now the data index is working in the same way
> > as
> > > > it did
> > > > > > > in
> > > > > > > > v7
> > > > > > > > > > > with
> > > > > > > > > > > > > the emitters. The only difference between two impl
> is
> > > > that in
> > > > > > > > here
> > > > > > > > > > the
> > > > > > > > > > > > > storage is pgsql instead elastic search.  You are
> > right
> > > > > > > regarding
> > > > > > > > > is
> > > > > > > > > > a
> > > > > > > > > > > > > snapshot of the last state of the process but we
> did
> > > > never
> > > > > > > define
> > > > > > > > > how
> > > > > > > > > > > > long
> > > > > > > > > > > > > would be alive that dats Honestly i am happy right
> > now
> > > > with
> > > > > > the
> > > > > > > > way
> > > > > > > > > > it
> > > > > > > > > > > > > works. The clean up mechanism is still tbd because
> we
> > > > still
> > > > > > > need
> > > > > > > > to
> > > > > > > > > > > > discuss
> > > > > > > > > > > > > other stuff first.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regarding stp is to leave no trail because u can
> get
> > > the
> > > > > > > outcome
> > > > > > > > > > > directly
> > > > > > > > > > > > > from the call. It was defined like that in v7. So
> > there
> > > > is no
> > > > > > > use
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > > index or the audit.
> > > > > > > > > > > > >
> > > > > > > > > > > > > El lun, 19 feb 2024, 14:13, Francisco Javier Tirado
> > > > Sarti <
> > > > > > > > > > > > > ftirados@redhat.com> escribió:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > > > > There has been some confusion about the purpose
> of
> > > > > > DataIndex.
> > > > > > > > To
> > > > > > > > > be
> > > > > > > > > > > > > honest
> > > > > > > > > > > > > > I believe they were already sorted out, but your
> > > e-mail
> > > > > > makes
> > > > > > > > me
> > > > > > > > > > > think
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > is not the case ;). I let Kris to clarify that
> with
> > > > you. My
> > > > > > > > view
> > > > > > > > > is
> > > > > > > > > > > > that
> > > > > > > > > > > > > > data-index is a way to query recently closed and
> > > active
> > > > > > > > processes
> > > > > > > > > > > (the
> > > > > > > > > > > > > key
> > > > > > > > > > > > > > here is the definition of recently, which in my
> > > opinion
> > > > > > > should
> > > > > > > > be
> > > > > > > > > > > > > > configurable)
> > > > > > > > > > > > > > But, besides that discussion and being pragmatic,
> > > > keeping
> > > > > > > > > finishing
> > > > > > > > > > > > > process
> > > > > > > > > > > > > > instances "for a while" in DataIndex was the only
> > way
> > > > for
> > > > > > > users
> > > > > > > > > to
> > > > > > > > > > > > query
> > > > > > > > > > > > > > the result of straight through processes. That's
> a
> > > > function
> > > > > > > > that
> > > > > > > > > > > cannot
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > removed right now
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Feb 19, 2024 at 1:33 PM Alex Porcelli <
> > > > > > > > > porcelli@apache.org
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > if data index was supposed to provide snapshot
> > view
> > > > of
> > > > > > the
> > > > > > > > > > process
> > > > > > > > > > > > > > > instance… why do we keep it after the process
> > > > instance is
> > > > > > > > > > finished?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, Feb 19, 2024 at 7:12 AM Francisco
> Javier
> > > > Tirado
> > > > > > > > Sarti <
> > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Martin.
> > > > > > > > > > > > > > > > After taking a deeper look at this, I realize
> > > that
> > > > the
> > > > > > > > > > behaviour
> > > > > > > > > > > is
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > expected one.
> > > > > > > > > > > > > > > > Runtimes DB does not track the completed
> > process
> > > > > > instance
> > > > > > > > > > (that's
> > > > > > > > > > > > > what
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > JDBCProcessInstances warn is telling us), but
> > > > > > DataIndex,
> > > > > > > as
> > > > > > > > > > > > expected,
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > tracking it in processes and nodes table. And
> > yes
> > > > it
> > > > > > will
> > > > > > > > > grow
> > > > > > > > > > > over
> > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > What we need is some configurable purge
> > mechanism
> > > > for
> > > > > > > > > > DataIndex,
> > > > > > > > > > > so
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > eventually removes older completed process
> > > > instances.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Feb 13, 2024 at 12:59 PM Francisco
> > Javier
> > > > > > Tirado
> > > > > > > > > Sarti
> > > > > > > > > > <
> > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Martin,
> > > > > > > > > > > > > > > > > Good catch!. Looks like the skipping
> > performed
> > > > for
> > > > > > > > process
> > > > > > > > > > > > > instances
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > not applied to node instances. Something we
> > > > > > definitely
> > > > > > > > need
> > > > > > > > > > to
> > > > > > > > > > > > > review
> > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > the runtimes side.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Mon, Feb 12, 2024 at 11:59 PM Martin
> > Weiler
> > > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >> On a somewhat related note, testing a
> simple
> > > > > > workflow
> > > > > > > > > (start
> > > > > > > > > > > ->
> > > > > > > > > > > > > > script
> > > > > > > > > > > > > > > > >> node -> end), I see the following messages
> > in
> > > > the
> > > > > > > logs:
> > > > > > > > > > > > > > > > >> 2024-02-12 22:49:50,493 28758dde544c WARN
> > > > > > > > > > > > > > > > >>
> > > > > > > > [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> > > > > > > > > > > > > > > > >> (executor-thread-3) Skipping create of
> > process
> > > > > > > instance
> > > > > > > > > id:
> > > > > > > > > > > > > > > > >> 7083088e-b899-47cb-b85c-5d9ccb0aa166,
> > state: 2
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> So far, so good. And I'd expect to see no
> > > trace
> > > > of
> > > > > > > this
> > > > > > > > > > > process
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > >> database if I don't have data audit
> enabled.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> However, the 'processes' table contains a
> > row
> > > > with
> > > > > > > > > state=2,
> > > > > > > > > > > with
> > > > > > > > > > > > > > > related
> > > > > > > > > > > > > > > > >> entries in the 'nodes' table. In a load
> > test,
> > > I
> > > > see
> > > > > > > > these
> > > > > > > > > > > tables
> > > > > > > > > > > > > > grow
> > > > > > > > > > > > > > > > >> significantly over time. Am I missing
> > > something
> > > > to
> > > > > > > have
> > > > > > > > > > these
> > > > > > > > > > > > > > entries
> > > > > > > > > > > > > > > > >> cleaned up automatically?
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> ________________________________________
> > > > > > > > > > > > > > > > >> From: Martin Weiler
> <mweiler@ibm.com.INVALID
> > >
> > > > > > > > > > > > > > > > >> Sent: Monday, February 12, 2024 3:40 PM
> > > > > > > > > > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > > > > > > > > > >> Subject: [EXTERNAL] RE: [DISCUSSION]
> > > Performance
> > > > > > > issues
> > > > > > > > > with
> > > > > > > > > > > > > > > data-index
> > > > > > > > > > > > > > > > >> persistence addon
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Thanks everyone for your input. Based on
> > this
> > > > > > > > discussion,
> > > > > > > > > I
> > > > > > > > > > > > opened
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > >> following PR:
> > > > > > > > > > > > > > > > >>
> > > > > > > > > >
> > > https://github.com/apache/incubator-kie-kogito-apps/pull/1985
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> With this change, the performance seems to
> > be
> > > > stable
> > > > > > > > over
> > > > > > > > > > > time:
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Martin
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> ________________________________________
> > > > > > > > > > > > > > > > >> From: Gonzalo Muñoz <gm...@apache.org>
> > > > > > > > > > > > > > > > >> Sent: Friday, February 9, 2024 9:42 AM
> > > > > > > > > > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > > > > > > > > > >> Subject: [EXTERNAL] Re: [DISCUSSION]
> > > Performance
> > > > > > > issues
> > > > > > > > > with
> > > > > > > > > > > > > > > data-index
> > > > > > > > > > > > > > > > >> persistence addon
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Great work Francisco,
> > > > > > > > > > > > > > > > >> Martin, take a look at this link with some
> > > > related
> > > > > > > tips
> > > > > > > > > (in
> > > > > > > > > > > case
> > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > find
> > > > > > > > > > > > > > > > >> it useful):
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > >
> > > > https://www.cybertec-postgresql.com/en/index-your-foreign-key/
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> El vie, 9 feb 2024 a las 17:20, Francisco
> > > Javier
> > > > > > > Tirado
> > > > > > > > > > Sarti
> > > > > > > > > > > (<
> > > > > > > > > > > > > > > > >> ftirados@redhat.com>) escribió:
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> > For the moment being, we will keep JPA
> > till
> > > we
> > > > > > > exhaust
> > > > > > > > > all
> > > > > > > > > > > > > > > > >> possibilities,
> > > > > > > > > > > > > > > > >> > let's call switching from jpa to jdbc
> our
> > > > hidden
> > > > > > > plan
> > > > > > > > B
> > > > > > > > > ;)
> > > > > > > > > > > > > > > > >> > I already told Martin, but in order
> > everyone
> > > > to
> > > > > > > know,
> > > > > > > > > just
> > > > > > > > > > > > after
> > > > > > > > > > > > > > > > writing
> > > > > > > > > > > > > > > > >> > the previous email, I thought "what if
> > > > Postgres is
> > > > > > > not
> > > > > > > > > > > > > > automatically
> > > > > > > > > > > > > > > > >> > indexing foreign keys like mysql?" and,
> > > eureka
> > > > > > > > > > > > > > > > >> > Postgres doc
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > >
> > > https://www.postgresql.org/docs/current/ddl-constraints.html
> > > > > > > > > > > > > > > > >> > Mysql doc
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >
> > > > > > > > > >
> > > > > >
> > https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> > > > > > > > > > > > > > > > >> > These are the relevant excerpt
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >> > *Postgresql*
> > > > > > > > > > > > > > > > >> > *A foreign key must reference columns
> that
> > > > either
> > > > > > > are
> > > > > > > > a
> > > > > > > > > > > > primary
> > > > > > > > > > > > > > key
> > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > >> form
> > > > > > > > > > > > > > > > >> > a unique constraint, or are columns
> from a
> > > > > > > non-partial
> > > > > > > > > > > unique
> > > > > > > > > > > > > > index.
> > > > > > > > > > > > > > > > >> This
> > > > > > > > > > > > > > > > >> > means that the referenced columns always
> > > have
> > > > an
> > > > > > > index
> > > > > > > > > to
> > > > > > > > > > > > allow
> > > > > > > > > > > > > > > > >> efficient
> > > > > > > > > > > > > > > > >> > lookups on whether a referencing row
> has a
> > > > match.
> > > > > > > > Since
> > > > > > > > > a
> > > > > > > > > > > > DELETE
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > >> row
> > > > > > > > > > > > > > > > >> > from the referenced table or an UPDATE
> of
> > a
> > > > > > > referenced
> > > > > > > > > > > column
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > >> require
> > > > > > > > > > > > > > > > >> > a scan of the referencing table for rows
> > > > matching
> > > > > > > the
> > > > > > > > > old
> > > > > > > > > > > > value,
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > >> > often a good idea to index the
> referencing
> > > > columns
> > > > > > > > too.
> > > > > > > > > > > > Because
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > >> not
> > > > > > > > > > > > > > > > >> > always needed, and there are many
> choices
> > > > > > available
> > > > > > > on
> > > > > > > > > how
> > > > > > > > > > > to
> > > > > > > > > > > > > > index,
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > >> > declaration of a foreign key constraint
> > does
> > > > not
> > > > > > > > > > > automatically
> > > > > > > > > > > > > > > create
> > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > >> > index on the referencing columns.*
> > > > > > > > > > > > > > > > >> > *Mysql*
> > > > > > > > > > > > > > > > >> > *MySQL requires that foreign key columns
> > be
> > > > > > indexed;
> > > > > > > > if
> > > > > > > > > > you
> > > > > > > > > > > > > > create a
> > > > > > > > > > > > > > > > >> table
> > > > > > > > > > > > > > > > >> > with a foreign key constraint but no
> index
> > > on
> > > > a
> > > > > > > given
> > > > > > > > > > > column,
> > > > > > > > > > > > an
> > > > > > > > > > > > > > > index
> > > > > > > > > > > > > > > > >> is
> > > > > > > > > > > > > > > > >> > created. *
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >> > So I asked Martin to especially create
> an
> > > > index
> > > > > > for
> > > > > > > > > > > > > > > > process_instance_id
> > > > > > > > > > > > > > > > >> > column on nodes table
> > > > > > > > > > > > > > > > >> > I think that will fix the problem
> detected
> > > on
> > > > the
> > > > > > > > thread
> > > > > > > > > > > dump.
> > > > > > > > > > > > > > > > >> > The simpler process test to verify
> queries
> > > are
> > > > > > fine
> > > > > > > > > still
> > > > > > > > > > > > > stands,
> > > > > > > > > > > > > > > > >> though ;)
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >> > On Fri, Feb 9, 2024 at 5:10 PM Tibor
> > > Zimányi <
> > > > > > > > > > > > > tzimanyi@apache.org
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >> > > I always preferred pure JDBC over
> > > Hibernate
> > > > > > > myself,
> > > > > > > > > just
> > > > > > > > > > > for
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > sake
> > > > > > > > > > > > > > > > >> of
> > > > > > > > > > > > > > > > >> > > control of what is happening :) So I
> > would
> > > > not
> > > > > > -1
> > > > > > > > that
> > > > > > > > > > > > myself.
> > > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > > >> > > Tibor
> > > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > > >> > > Dňa pi 9. 2. 2024, 17:00 Francisco
> > Javier
> > > > Tirado
> > > > > > > > > Sarti <
> > > > > > > > > > > > > > > > >> > > ftirados@redhat.com>
> > > > > > > > > > > > > > > > >> > > napísal(a):
> > > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > > >> > > > Hi,
> > > > > > > > > > > > > > > > >> > > > Usually I do not want to talk about
> > work
> > > > in
> > > > > > > > progress
> > > > > > > > > > > > because
> > > > > > > > > > > > > > > > >> > preliminary
> > > > > > > > > > > > > > > > >> > > > conclusions are pretty volatile but,
> > > well,
> > > > > > there
> > > > > > > > > are a
> > > > > > > > > > > > > couple
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > >> things
> > > > > > > > > > > > > > > > >> > > > that can be concluded from the
> really
> > > > valuable
> > > > > > > > > > > information
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > >> Martin
> > > > > > > > > > > > > > > > >> > > > provided.
> > > > > > > > > > > > > > > > >> > > > 1) In order to be able to determine
> if
> > > the
> > > > > > > number
> > > > > > > > of
> > > > > > > > > > > > > > statements
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > >> > larger
> > > > > > > > > > > > > > > > >> > > > than expected, I asked Martin to
> test
> > > > with a
> > > > > > > > simpler
> > > > > > > > > > > > process
> > > > > > > > > > > > > > > > >> > definition.
> > > > > > > > > > > > > > > > >> > > > One with just three nodes: start,
> > script
> > > > and
> > > > > > > end.
> > > > > > > > > The
> > > > > > > > > > > > script
> > > > > > > > > > > > > > one
> > > > > > > > > > > > > > > > >> should
> > > > > > > > > > > > > > > > >> > > > change just one variable. This way
> we
> > > can
> > > > > > > analyze
> > > > > > > > if
> > > > > > > > > > the
> > > > > > > > > > > > > > number
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > >> > > queries
> > > > > > > > > > > > > > > > >> > > > is the expected one. From the single
> > log
> > > > > > (audit
> > > > > > > > was
> > > > > > > > > > > > > activated
> > > > > > > > > > > > > > > > them)
> > > > > > > > > > > > > > > > >> my
> > > > > > > > > > > > > > > > >> > > > conclusion is that the number of
> > > > > > insert/updates
> > > > > > > > over
> > > > > > > > > > > > > processes
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > >> > nodes
> > > > > > > > > > > > > > > > >> > > > (there a lot over task, that I will
> > > > prefer to
> > > > > > > skip
> > > > > > > > > for
> > > > > > > > > > > > now,
> > > > > > > > > > > > > > baby
> > > > > > > > > > > > > > > > >> steps)
> > > > > > > > > > > > > > > > >> > > is
> > > > > > > > > > > > > > > > >> > > > the expected one.
> > > > > > > > > > > > > > > > >> > > > 2) Analysing the thread dump, we see
> > > > around 15
> > > > > > > > > threads
> > > > > > > > > > > > > > executing
> > > > > > > > > > > > > > > > >> this
> > > > > > > > > > > > > > > > >> > > line
> > > > > > > > > > > > > > > > >> > > > at
> > > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > > > > > > > > > > > > > > >> > > > so its pretty clear the code to be
> > > > optimized
> > > > > > ;).
> > > > > > > > I'm
> > > > > > > > > > > > > > evaluating
> > > > > > > > > > > > > > > > >> > > > possibilities within JPA/Hibernate,
> > but
> > > > I'm
> > > > > > > > starting
> > > > > > > > > > to
> > > > > > > > > > > > > think
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > >> it
> > > > > > > > > > > > > > > > >> > > might
> > > > > > > > > > > > > > > > >> > > > be better to switch to JDBC and skip
> > > > > > hibernate.
> > > > > > > > Our
> > > > > > > > > > > lives
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > >> > > simpler,
> > > > > > > > > > > > > > > > >> > > > especially with a schema relatively
> > > simple
> > > > > > like
> > > > > > > > ours
> > > > > > > > > > > (that
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > >> my
> > > > > > > > > > > > > > > > >> > > > recommendation if I was an external
> > > > > > consultant)
> > > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > > >> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor
> > > > Zimányi <
> > > > > > > > > > > > > > > tzimanyi@apache.org
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > > >> > > > > Hi,
> > > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > > >> > > > > this will be a bit off-topic.
> > However
> > > > as far
> > > > > > > as
> > > > > > > > > > > > > > performance, I
> > > > > > > > > > > > > > > > >> think
> > > > > > > > > > > > > > > > >> > we
> > > > > > > > > > > > > > > > >> > > > > should think about that we have
> > string
> > > > > > primary
> > > > > > > > > keys
> > > > > > > > > > > > > (IDs). I
> > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > >> > > expect
> > > > > > > > > > > > > > > > >> > > > > the database systems are much
> better
> > > > with
> > > > > > > > indexing
> > > > > > > > > > > > numeric
> > > > > > > > > > > > > > > keys
> > > > > > > > > > > > > > > > >> than
> > > > > > > > > > > > > > > > >> > > > > strings. I remember from the past,
> > > when
> > > > I
> > > > > > was
> > > > > > > > > > working
> > > > > > > > > > > > with
> > > > > > > > > > > > > > > DBs,
> > > > > > > > > > > > > > > > >> that
> > > > > > > > > > > > > > > > >> > > > using
> > > > > > > > > > > > > > > > >> > > > > strings as keys or indexes was a
> > > > discouraged
> > > > > > > > > > practice.
> > > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > > >> > > > > Best regards,
> > > > > > > > > > > > > > > > >> > > > > Tibor
> > > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > > >> > > > > Dňa št 8. 2. 2024, 22:45 Martin
> > Weiler
> > > > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >> > > > > napísal(a):
> > > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > > >> > > > > > I changed the test to use
> MongoDB
> > > [1]
> > > > and
> > > > > > I
> > > > > > > > > don't
> > > > > > > > > > > see
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > > >> performance
> > > > > > > > > > > > > > > > >> > > > > > degradation with this setup [2].
> > > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > > >> > > > > > Please keep us posted of your
> > > > findings.
> > > > > > > > Thanks!
> > > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > > >> > > > > > Martin
> > > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > > >> > > > > > [1]
> > > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > >
> https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > > > > > > > > > > > > > >> > > > > > [2]
> > > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > > >> > > > > >
> > > > ________________________________________
> > > > > > > > > > > > > > > > >> > > > > > From: Francisco Javier Tirado
> > Sarti
> > > <
> > > > > > > > > > > > > ftirados@redhat.com>
> > > > > > > > > > > > > > > > >> > > > > > Sent: Wednesday, February 7,
> 2024
> > > > 11:40 AM
> > > > > > > > > > > > > > > > >> > > > > > To: dev@kie.apache.org
> > > > > > > > > > > > > > > > >> > > > > > Subject: [EXTERNAL] Re:
> > [DISCUSSION]
> > > > > > > > Performance
> > > > > > > > > > > > issues
> > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > >> > > data-index
> > > > > > > > > > > > > > > > >> > > > > > persistence addon
> > > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > > >> > > > > > yes, it can be index degradation
> > > > because
> > > > > > of
> > > > > > > > > size,
> > > > > > > > > > > but
> > > > > > > > > > > > I
> > > > > > > > > > > > > > > > believe
> > > > > > > > > > > > > > > > >> (I
> > > > > > > > > > > > > > > > >> > > > might
> > > > > > > > > > > > > > > > >> > > > > be
> > > > > > > > > > > > > > > > >> > > > > > wrong) the db is too small (yet)
> > for
> > > > that.
> > > > > > > > > > > > > > > > >> > > > > > But, eventually, Postgres, when
> > the
> > > > DB is
> > > > > > > huge
> > > > > > > > > > > enough,
> > > > > > > > > > > > > > > > >> unavoidably
> > > > > > > > > > > > > > > > >> > > will
> > > > > > > > > > > > > > > > >> > > > > > behave like the graphic that
> > Martin
> > > > sent.
> > > > > > > > > > > > > > > > >> > > > > > Since I believe we are not huge
> > > enough
> > > > > > > (yet),
> > > > > > > > > lets
> > > > > > > > > > > > rule
> > > > > > > > > > > > > > out
> > > > > > > > > > > > > > > > >> another
> > > > > > > > > > > > > > > > >> > > > issue
> > > > > > > > > > > > > > > > >> > > > > > by analysing the sql logs (I
> > > requested
> > > > > > those
> > > > > > > > to
> > > > > > > > > > > Martin
> > > > > > > > > > > > > > > offline
> > > > > > > > > > > > > > > > >> and
> > > > > > > > > > > > > > > > >> > he
> > > > > > > > > > > > > > > > >> > > > is
> > > > > > > > > > > > > > > > >> > > > > > going to kindly collect them).
> > > > > > > > > > > > > > > > >> > > > > > Also Im curious to know if Mongo
> > > > behave in
> > > > > > > the
> > > > > > > > > > same
> > > > > > > > > > > > way.
> > > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > > >> > > > > > On Wed, Feb 7, 2024 at 7:25 PM
> > > Enrique
> > > > > > > > Gonzalez
> > > > > > > > > > > > > Martinez <
> > > > > > > > > > > > > > > > >> > > > > > egonzalez@apache.org> wrote:
> > > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > > >> > > > > > > Hi Francisco,
> > > > > > > > > > > > > > > > >> > > > > > > I would highly recommend to
> > check
> > > > > > indexes
> > > > > > > > and
> > > > > > > > > > how
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > updates
> > > > > > > > > > > > > > > > >> > work
> > > > > > > > > > > > > > > > >> > > in
> > > > > > > > > > > > > > > > >> > > > > > data
> > > > > > > > > > > > > > > > >> > > > > > > index to avoid full scan table
> > and
> > > > lock
> > > > > > > the
> > > > > > > > > full
> > > > > > > > > > > > > table.
> > > > > > > > > > > > > > > Some
> > > > > > > > > > > > > > > > >> db
> > > > > > > > > > > > > > > > >> > are
> > > > > > > > > > > > > > > > >> > > > > very
> > > > > > > > > > > > > > > > >> > > > > > > sensitive to that.
> > > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > > >> > > > > > > El mié, 7 feb 2024, 18:41,
> > > Francisco
> > > > > > > Javier
> > > > > > > > > > Tirado
> > > > > > > > > > > > > > Sarti <
> > > > > > > > > > > > > > > > >> > > > > > > ftirados@redhat.com>
> escribió:
> > > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > > >> > > > > > > > Hi Martin,
> > > > > > > > > > > > > > > > >> > > > > > > > While I analyze the data,
> let
> > me
> > > > ask
> > > > > > you
> > > > > > > > if
> > > > > > > > > it
> > > > > > > > > > > is
> > > > > > > > > > > > > > > possible
> > > > > > > > > > > > > > > > >> to
> > > > > > > > > > > > > > > > >> > > > perform
> > > > > > > > > > > > > > > > >> > > > > > > > another check (similar in a
> > way
> > > to
> > > > > > > > disabling
> > > > > > > > > > > > > > data-index
> > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > >> > you
> > > > > > > > > > > > > > > > >> > > > do)
> > > > > > > > > > > > > > > > >> > > > > > Can
> > > > > > > > > > > > > > > > >> > > > > > > > you switch to MongoDB
> > > persistence
> > > > and
> > > > > > > > check
> > > > > > > > > if
> > > > > > > > > > > the
> > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > >> > > degradation
> > > > > > > > > > > > > > > > >> > > > > > that
> > > > > > > > > > > > > > > > >> > > > > > > is
> > > > > > > > > > > > > > > > >> > > > > > > > there for postgres remains?
> > > > > > > > > > > > > > > > >> > > > > > > > I do not know if this is
> > > feasible
> > > > but
> > > > > > > will
> > > > > > > > > > > > certainly
> > > > > > > > > > > > > > > > >> indicate
> > > > > > > > > > > > > > > > >> > the
> > > > > > > > > > > > > > > > >> > > > > > problem
> > > > > > > > > > > > > > > > >> > > > > > > > is on the postgres storage
> > layer
> > > > and I
> > > > > > > do
> > > > > > > > > not
> > > > > > > > > > > > have a
> > > > > > > > > > > > > > > clear
> > > > > > > > > > > > > > > > >> > > > prediction
> > > > > > > > > > > > > > > > >> > > > > > of
> > > > > > > > > > > > > > > > >> > > > > > > > what we will see when doing
> > this
> > > > > > switch.
> > > > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > > > >> > > > > > > > On Wed, Feb 7, 2024 at
> 6:37 PM
> > > > Martin
> > > > > > > > Weiler
> > > > > > > > > > > > > > > > >> > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > > >> > > > > > > > wrote:
> > > > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > > > >> > > > > > > > > Hi Francisco,
> > > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > > >> > > > > > > > > thanks for your work on
> this
> > > > > > important
> > > > > > > > > > topic!
> > > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > > >> > > > > > > > > I would like to share some
> > > test
> > > > > > > results
> > > > > > > > > > here,
> > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > >> > help
> > > > > > > > > > > > > > > > >> > > to
> > > > > > > > > > > > > > > > >> > > > > > > improve
> > > > > > > > > > > > > > > > >> > > > > > > > > the codebase even
> further. I
> > > am
> > > > > > using
> > > > > > > > the
> > > > > > > > > > > jmeter
> > > > > > > > > > > > > > based
> > > > > > > > > > > > > > > > >> test
> > > > > > > > > > > > > > > > >> > > case
> > > > > > > > > > > > > > > > >> > > > > from
> > > > > > > > > > > > > > > > >> > > > > > > > Pere
> > > > > > > > > > > > > > > > >> > > > > > > > > and Enrique (thanks guys!)
> > [1]
> > > > which
> > > > > > > > uses
> > > > > > > > > a
> > > > > > > > > > > load
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > 30
> > > > > > > > > > > > > > > > >> > threads
> > > > > > > > > > > > > > > > >> > > to
> > > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > > >> > > > > > > > > 1) start a new process
> > > instance
> > > > > > (POST)
> > > > > > > > > > > > > > > > >> > > > > > > > > 2) retrieve tasks for a
> user
> > > > (GET)
> > > > > > > > > > > > > > > > >> > > > > > > > > 3) fetches task details
> > (GET)
> > > > > > > > > > > > > > > > >> > > > > > > > > 4) complete a task (POST)
> > > > > > > > > > > > > > > > >> > > > > > > > > 5) execute a query on
> > > data-audit
> > > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > > >> > > > > > > > > With this test setup, I
> > > noticed
> > > > that
> > > > > > > the
> > > > > > > > > > > > > performance
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > > > >> > > POST
> > > > > > > > > > > > > > > > >> > > > > > > > > requests, in particular
> the
> > > one
> > > > to
> > > > > > > > start a
> > > > > > > > > > new
> > > > > > > > > > > > > > process
> > > > > > > > > > > > > > > > >> > > instance,
> > > > > > > > > > > > > > > > >> > > > > > > degrades
> > > > > > > > > > > > > > > > >> > > > > > > > > over time - see graph [2].
> > If
> > > I
> > > > run
> > > > > > > the
> > > > > > > > > same
> > > > > > > > > > > > test
> > > > > > > > > > > > > > > > without
> > > > > > > > > > > > > > > > >> > > > > data-index,
> > > > > > > > > > > > > > > > >> > > > > > > > then
> > > > > > > > > > > > > > > > >> > > > > > > > > there is no such
> performance
> > > > > > > degradation
> > > > > > > > > > [3].
> > > > > > > > > > > > You
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > >> find a
> > > > > > > > > > > > > > > > >> > > > thread
> > > > > > > > > > > > > > > > >> > > > > > > dump
> > > > > > > > > > > > > > > > >> > > > > > > > > captured a few minutes
> into
> > > the
> > > > > > first
> > > > > > > > test
> > > > > > > > > > > here
> > > > > > > > > > > > > [4]
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > >> > might
> > > > > > > > > > > > > > > > >> > > > help
> > > > > > > > > > > > > > > > >> > > > > > to
> > > > > > > > > > > > > > > > >> > > > > > > > see
> > > > > > > > > > > > > > > > >> > > > > > > > > some of the contention
> > points.
> > > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > > >> > > > > > > > > I'd appreciate if you
> could
> > > > take a
> > > > > > > look
> > > > > > > > > and
> > > > > > > > > > > see
> > > > > > > > > > > > if
> > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > >> is
> > > > > > > > > > > > > > > > >> > > > > something
> > > > > > > > > > > > > > > > >> > > > > > > > that
> > > > > > > > > > > > > > > > >> > > > > > > > > can be further improved
> > based
> > > on
> > > > > > your
> > > > > > > > > > previous
> > > > > > > > > > > > > work.
> > > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > >> you
> > > > > > > > > > > > > > > > >> > > need
> > > > > > > > > > > > > > > > >> > > > > any
> > > > > > > > > > > > > > > > >> > > > > > > > > additional data, let me
> > know,
> > > > but
> > > > > > > > > otherwise
> > > > > > > > > > it
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > >> > > straightforward
> > > > > > > > > > > > > > > > >> > > > > to
> > > > > > > > > > > > > > > > >> > > > > > > run
> > > > > > > > > > > > > > > > >> > > > > > > > > the jmeter test as well.
> > > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > > >> > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > >> > > > > > > > > Martin
> > > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > > >> > > > > > > > > [1]
> > > > > > > > > > > > > > > > >>
> > > > > > > https://github.com/pefernan/job-service-refactor-test/
> > > > > > > > > > > > > > > > >> > > > > > > > > [2]
> > > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > > > > > > > > > > > > >> > > > > > > > > [3]
> > > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > > > > > > > > > > > > >> > > > > > > > > [4]
> > > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > ________________________________________
> > > > > > > > > > > > > > > > >> > > > > > > > > From: Francisco Javier
> > Tirado
> > > > Sarti
> > > > > > <
> > > > > > > > > > > > > > > > ftirados@redhat.com>
> > > > > > > > > > > > > > > > >> > > > > > > > > Sent: Wednesday, January
> 17,
> > > > 2024
> > > > > > 9:13
> > > > > > > > AM
> > > > > > > > > > > > > > > > >> > > > > > > > > To: dev@kie.apache.org
> > > > > > > > > > > > > > > > >> > > > > > > > > Cc: Pere Fernandez Perez
> > > > > > > > > > > > > > > > >> > > > > > > > > Subject: [EXTERNAL] Re:
> > > > [DISCUSSION]
> > > > > > > > > > > Performance
> > > > > > > > > > > > > > > issues
> > > > > > > > > > > > > > > > >> with
> > > > > > > > > > > > > > > > >> > > > > > data-index
> > > > > > > > > > > > > > > > >> > > > > > > > > persistence addon
> > > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > > >> > > > > > > > > Hi Alex,
> > > > > > > > > > > > > > > > >> > > > > > > > > I did not take times
> (which
> > > > depends
> > > > > > > on a
> > > > > > > > > > > number
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > >> variables
> > > > > > > > > > > > > > > > >> > > that
> > > > > > > > > > > > > > > > >> > > > > > > > > drastically change between
> > > > > > > > environments),
> > > > > > > > > > but
> > > > > > > > > > > > > verify
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > > > >> > > > > number
> > > > > > > > > > > > > > > > >> > > > > > of
> > > > > > > > > > > > > > > > >> > > > > > > > > updates has been reduced
> > > > drastically
> > > > > > > > > without
> > > > > > > > > > > > > losing
> > > > > > > > > > > > > > > > >> > > > functionality,
> > > > > > > > > > > > > > > > >> > > > > > > which
> > > > > > > > > > > > > > > > >> > > > > > > > is
> > > > > > > > > > > > > > > > >> > > > > > > > > objectively a good thing.
> If
> > > > before
> > > > > > > the
> > > > > > > > > > > change,
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > every
> > > > > > > > > > > > > > > > >> > node
> > > > > > > > > > > > > > > > >> > > > > > > executed,
> > > > > > > > > > > > > > > > >> > > > > > > > we
> > > > > > > > > > > > > > > > >> > > > > > > > > have an update for every
> > node
> > > > > > > previously
> > > > > > > > > > > > executed,
> > > > > > > > > > > > > > so
> > > > > > > > > > > > > > > > if a
> > > > > > > > > > > > > > > > >> > > > process
> > > > > > > > > > > > > > > > >> > > > > > have
> > > > > > > > > > > > > > > > >> > > > > > > > 50
> > > > > > > > > > > > > > > > >> > > > > > > > > nodes to execute, we were
> > > > performing
> > > > > > > > > nearly
> > > > > > > > > > > > > 50*51/2
> > > > > > > > > > > > > > > > >> updates,
> > > > > > > > > > > > > > > > >> > > > which
> > > > > > > > > > > > > > > > >> > > > > > > gives
> > > > > > > > > > > > > > > > >> > > > > > > > us
> > > > > > > > > > > > > > > > >> > > > > > > > > a total of  1275 updates,
> > now
> > > we
> > > > > > have
> > > > > > > > just
> > > > > > > > > > one
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > every
> > > > > > > > > > > > > > > > >> node
> > > > > > > > > > > > > > > > >> > > > being
> > > > > > > > > > > > > > > > >> > > > > > > > > executed, implying a total
> > of
> > > 50
> >
>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Alex Porcelli <po...@apache.org>]

Francisco,

We wont use cron ourselves, we’ll recommend to users if they want a regular
cleanup. We don’t need to own this, Kubernetes has native cron and OSs also
have their native cron…. So we don’t need replicate such functionality.

I think the cleanup we are talking about here is a bulk operation,
individual instances deletion should be handled differently.

On Thu, Feb 22, 2024 at 6:03 AM Francisco Javier Tirado Sarti <
ftirados@redhat.com> wrote:

> Ok, so we have two flavours, clean up on demand through REST user call
> (that's fine, the only issue is the scope of it. For example, should we
> allow a bulk delete api where you specify the process instances, if the
> user knows them somehow, that user wants to delete or just, as we are
> proposing, delete all instances that have been there for a while?) and cron
> process (that is a replacement of the properties approach I recommend).
> In my opinion, if we have the explicit REST API, we should not use the
> cron.
>
> On Thu, Feb 22, 2024 at 6:32 AM Enrique Gonzalez Martinez <
> egonzalez@apache.org> wrote:
>
> > +1 alex
> >
> > El jue, 22 feb 2024, 0:14, Alex Porcelli <al...@porcelli.me> escribió:
> >
> > > I think I need to clarify what I mean by REST...
> > >
> > > The REST endpoint I have in mind would execute the necessary cleaning
> > > based on an input parameter (ie. instanced closed for more than X days
> > > or something like that).
> > >
> > > The use of cron is that an external cron could call the cleaning REST
> > > endpoint from time to time (once a month, on every sunday night, etc).
> > >
> > >
> > > On Wed, Feb 21, 2024 at 10:50 AM Francisco Javier Tirado Sarti
> > > <ft...@redhat.com> wrote:
> > > >
> > > > Hi Alex,
> > > > I'm wondering how cron and rest can live together.
> > > > With cron the responsibility to do the clean up relies on a process
> > > > external to the microservice accessing the db to be purged, while
> where
> > > > using REST API the process performing the clean up is typically (but
> > not
> > > > necessarily) the same microservice that accesses the DB. In that
> sense,
> > > > REST API is similar to using properties (in both approaches the
> > > responsible
> > > > to clean up the data is the microservice), but they are different
> > because
> > > > REST is more sophisticated (but compatible, implementation wise we
> can
> > > use
> > > > a ConfigSource, that is updated by the REST and read, as a property,
> by
> > > the
> > > > cleanup routine). So, probably I'm missing something but I see the
> cron
> > > > approach not reconcilable with the REST approach.
> > > > Anyway just a detail, I agree the purge approach should be similar
> for
> > > all
> > > > microservices
> > > >
> > > > On Wed, Feb 21, 2024 at 4:37 PM Alex Porcelli <po...@apache.org>
> > > wrote:
> > > >
> > > > > +1 for provide consistent behavior across the board
> > > > >
> > > > > my 2c:
> > > > >
> > > > > +1 for REST interface (preferred for external general management
> > > > > perspective)
> > > > > -1 for properties to auto clean; this can be achieved with cron and
> > > rest
> > > > > call
> > > > >
> > > > > Now about GraphQL: the platform today is heavily invested in such
> > > > > interface, so it might make sense. BUT I’d consider it in a second
> > > moment.
> > > > >
> > > > >
> > > > > On Wed, Feb 21, 2024 at 6:52 AM Francisco Javier Tirado Sarti <
> > > > > ftirados@redhat.com> wrote:
> > > > >
> > > > > > Hi Enrique,
> > > > > > If we configure such policies through properties, it won't be
> > enough
> > > to
> > > > > > define a naming convention and rely on the target platform
> > > configuration
> > > > > > capabilities (SpringBoot or Quarkus)?.
> > > > > >
> > > > > > On Wed, Feb 21, 2024 at 12:36 PM Enrique Gonzalez Martinez <
> > > > > > egonzalez@apache.org> wrote:
> > > > > >
> > > > > > > Hi Francisco
> > > > > > >
> > > > > > > The discussion we need to have before how to achieve certain
> > > features,
> > > > > is
> > > > > > > the overall of the user experience. If you want to clean it up
> > > that way
> > > > > > it
> > > > > > > is fine by me. I have nothing against setting policy related to
> > > some
> > > > > sort
> > > > > > > of clean up for a process completed some time ago.
> > > > > > >
> > > > > > > As we discussed previously the data index is a snapshot of the
> > last
> > > > > state
> > > > > > > of the process instance included completed, but nothing was
> said
> > > once
> > > > > > > completed when we need to clean that up so any policy is
> welcome.
> > > How
> > > > > to
> > > > > > > achieve that policy is something completely different.
> > > > > > >
> > > > > > > The main problem is how every microservice is exposing that API
> > to
> > > the
> > > > > > end
> > > > > > > user or consumer (other system) and which system is going to be
> > the
> > > > > > façade
> > > > > > > for complex deployments and operations. That is the discussion
> I
> > > was
> > > > > > > mentioning before.
> > > > > > >
> > > > > > > What I want to avoid is to set different policies among
> > components
> > > and
> > > > > we
> > > > > > > should strive to be as much as possible to offer certain
> > > capabilities
> > > > > in
> > > > > > > the same fashion, e.g: clean up mechanism.
> > > > > > > In the same way I want to avoid making the current system more
> > > complex
> > > > > > than
> > > > > > > they are. So far they are aligned offering one simple
> > > responsibility
> > > > > but
> > > > > > I
> > > > > > > would be against mixing for instance job service with data
> index.
> > > > > > >
> > > > > > > El mié, 21 feb 2024 a las 12:13, Francisco Javier Tirado Sarti
> (<
> > > > > > > ftirados@redhat.com>) escribió:
> > > > > > >
> > > > > > > > Hi Enrique,
> > > > > > > > In the case of data index I think the data to be purged is
> > > finished
> > > > > > > process
> > > > > > > > instances (I do not think we should remove process instance
> > that
> > > has
> > > > > > been
> > > > > > > > alive for ages, even if it is very likely they are not going
> to
> > > be
> > > > > ever
> > > > > > > > completed)
> > > > > > > > Once you delete those process instances, you also delete the
> > > > > associated
> > > > > > > > user tasks and jobs.
> > > > > > > > Therefore the problem is relatively simple, to be able to
> > > configure
> > > > > how
> > > > > > > > much time a completed process instance should remain in the
> > data
> > > > > index
> > > > > > > > database. We can take a simple approach: a property with a
> min
> > > > > duration
> > > > > > > > that cannot be changed once data index is started; a slightly
> > > complex
> > > > > > > one:
> > > > > > > > the same property but watching it to react for changes; or
> the
> > > full
> > > > > > > suite:
> > > > > > > > an admin API to be able to change the policy at any moment.
> > > > > > > > I think this discussion is worth having.
> > > > > > > >
> > > > > > > > On Wed, Feb 21, 2024 at 6:15 AM Enrique Gonzalez Martinez <
> > > > > > > > egonzalez@apache.org> wrote:
> > > > > > > >
> > > > > > > > > Hi Martin, the main problem regarding the purge is because
> it
> > > is
> > > > > > still
> > > > > > > > > unclear the policy for what tech to use and the future
> > > components.
> > > > > > > > >
> > > > > > > > > Recently we had a discussion about proposing graphql for
> this
> > > sort
> > > > > of
> > > > > > > > admin
> > > > > > > > > tasks. So far for subsystems we have been using rest
> > endpoints
> > > > > (like
> > > > > > > > update
> > > > > > > > > timers or modify human task or change processes). There is
> > one
> > > > > > > exception
> > > > > > > > > which is the gateway that is pure graphql and somehow uses
> > > graphql
> > > > > > for
> > > > > > > > > everything making complex operations under the hood.
> > > > > > > > >
> > > > > > > > > This has somehow frozen the purge for data audit for a bit
> > and
> > > the
> > > > > > > > proposal
> > > > > > > > > was to use rest endpoints to do the clean up in the
> component
> > > and
> > > > > > offer
> > > > > > > > the
> > > > > > > > > graphql counterpart in the gateway promoting it to a first
> > > class
> > > > > > > citizen
> > > > > > > > > component instead of having it embedded in the data index.
> > > > > > > > >
> > > > > > > > > I would suggest to come up at least with a policy first
> > > regarding
> > > > > the
> > > > > > > > > convention every component should address this.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > El mar, 20 feb 2024, 23:39, Martin Weiler
> > > <mweiler@ibm.com.invalid
> > > > > >
> > > > > > > > > escribió:
> > > > > > > > >
> > > > > > > > > > IMO, it is good to have this discussion around data
> sanity
> > > now
> > > > > > > instead
> > > > > > > > of
> > > > > > > > > > putting it off until later when data has already
> > accumulated
> > > in
> > > > > > > > > production
> > > > > > > > > > environments.
> > > > > > > > > >
> > > > > > > > > > Based on the input here, we are dealing with three types
> of
> > > data:
> > > > > > > > > > 1. Runtime data - active instances only, engine cleans up
> > the
> > > > > data
> > > > > > > > > > automatically at process instance end
> > > > > > > > > > 2. Historic log data - data created by data-audit
> intended
> > > for
> > > > > long
> > > > > > > > term
> > > > > > > > > > storage
> > > > > > > > > > 3. Data-index data - somehow this data falls in between
> the
> > > two
> > > > > > > > > > aforementioned categories, with the idea of the data
> being
> > > > > > "recent",
> > > > > > > > but
> > > > > > > > > > not restricted to active instances only
> > > > > > > > > >
> > > > > > > > > > We'd need purge strategies for both #2 and #3 (perhaps
> > > different
> > > > > > > ones,
> > > > > > > > or
> > > > > > > > > > with different config settings) in order to prevent
> > unlimited
> > > > > data
> > > > > > > > > growth.
> > > > > > > > > >
> > > > > > > > > > ________________________________________
> > > > > > > > > > From: Enrique Gonzalez Martinez <eg...@apache.org>
> > > > > > > > > > Sent: Monday, February 19, 2024 7:11 AM
> > > > > > > > > > To: dev@kie.apache.org
> > > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues
> > with
> > > > > > > data-index
> > > > > > > > > > persistence addon
> > > > > > > > > >
> > > > > > > > > > Hi Francisco,
> > > > > > > > > > To give you more context about this.
> > > > > > > > > >
> > > > > > > > > > STP is a concept, a process with certain constraints: no
> > > > > > persistence
> > > > > > > > and
> > > > > > > > > > returning the outcome in the call (sync execution with no
> > > idle
> > > > > > > states).
> > > > > > > > > It
> > > > > > > > > > was a requirement from a user in the past. One of the
> > > > > requirements
> > > > > > > was
> > > > > > > > > > leaving no trail. In v7 was easy because you could
> disable
> > > the
> > > > > > audit
> > > > > > > in
> > > > > > > > > > that case. Actually we have the same way to do what we
> did
> > > in v7
> > > > > in
> > > > > > > > here
> > > > > > > > > as
> > > > > > > > > > you can add/remove index just removing deps.
> > > > > > > > > >
> > > > > > > > > > We have the same outcome with different approaches and
> STP
> > is
> > > > > > already
> > > > > > > > > > delivered.
> > > > > > > > > >
> > > > > > > > > > El lun, 19 feb 2024 a las 14:46, Francisco Javier Tirado
> > > Sarti (<
> > > > > > > > > > ftirados@redhat.com>) escribió:
> > > > > > > > > >
> > > > > > > > > > > Regarding STP (which is not a concept that we have in
> the
> > > > > code. I
> > > > > > > > mean
> > > > > > > > > > STP
> > > > > > > > > > > are processes as nonSTP are), I guess, as all
> processes,
> > > they
> > > > > > were
> > > > > > > > kept
> > > > > > > > > > in
> > > > > > > > > > > DataIndex once completed because users wanted (and
> still
> > > wants)
> > > > > > to
> > > > > > > > > check
> > > > > > > > > > > the result once the call had been performed. If we want
> > to
> > > > > leave
> > > > > > no
> > > > > > > > > trace
> > > > > > > > > > > of them in DataIndex for some reason, we will need to
> > make
> > > it a
> > > > > > > > > > > Runtimes concept so DataIndex can handle them in a
> > > different
> > > > > way.
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Feb 19, 2024 at 2:27 PM Enrique Gonzalez
> > Martinez <
> > > > > > > > > > > egonzalez@apache.org> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Alex:
> > > > > > > > > > > > Right now the data index is working in the same way
> as
> > > it did
> > > > > > in
> > > > > > > v7
> > > > > > > > > > with
> > > > > > > > > > > > the emitters. The only difference between two impl is
> > > that in
> > > > > > > here
> > > > > > > > > the
> > > > > > > > > > > > storage is pgsql instead elastic search.  You are
> right
> > > > > > regarding
> > > > > > > > is
> > > > > > > > > a
> > > > > > > > > > > > snapshot of the last state of the process but we did
> > > never
> > > > > > define
> > > > > > > > how
> > > > > > > > > > > long
> > > > > > > > > > > > would be alive that dats Honestly i am happy right
> now
> > > with
> > > > > the
> > > > > > > way
> > > > > > > > > it
> > > > > > > > > > > > works. The clean up mechanism is still tbd because we
> > > still
> > > > > > need
> > > > > > > to
> > > > > > > > > > > discuss
> > > > > > > > > > > > other stuff first.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Regarding stp is to leave no trail because u can get
> > the
> > > > > > outcome
> > > > > > > > > > directly
> > > > > > > > > > > > from the call. It was defined like that in v7. So
> there
> > > is no
> > > > > > use
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > > index or the audit.
> > > > > > > > > > > >
> > > > > > > > > > > > El lun, 19 feb 2024, 14:13, Francisco Javier Tirado
> > > Sarti <
> > > > > > > > > > > > ftirados@redhat.com> escribió:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > > > There has been some confusion about the purpose of
> > > > > DataIndex.
> > > > > > > To
> > > > > > > > be
> > > > > > > > > > > > honest
> > > > > > > > > > > > > I believe they were already sorted out, but your
> > e-mail
> > > > > makes
> > > > > > > me
> > > > > > > > > > think
> > > > > > > > > > > > that
> > > > > > > > > > > > > is not the case ;). I let Kris to clarify that with
> > > you. My
> > > > > > > view
> > > > > > > > is
> > > > > > > > > > > that
> > > > > > > > > > > > > data-index is a way to query recently closed and
> > active
> > > > > > > processes
> > > > > > > > > > (the
> > > > > > > > > > > > key
> > > > > > > > > > > > > here is the definition of recently, which in my
> > opinion
> > > > > > should
> > > > > > > be
> > > > > > > > > > > > > configurable)
> > > > > > > > > > > > > But, besides that discussion and being pragmatic,
> > > keeping
> > > > > > > > finishing
> > > > > > > > > > > > process
> > > > > > > > > > > > > instances "for a while" in DataIndex was the only
> way
> > > for
> > > > > > users
> > > > > > > > to
> > > > > > > > > > > query
> > > > > > > > > > > > > the result of straight through processes. That's a
> > > function
> > > > > > > that
> > > > > > > > > > cannot
> > > > > > > > > > > > be
> > > > > > > > > > > > > removed right now
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Feb 19, 2024 at 1:33 PM Alex Porcelli <
> > > > > > > > porcelli@apache.org
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > if data index was supposed to provide snapshot
> view
> > > of
> > > > > the
> > > > > > > > > process
> > > > > > > > > > > > > > instance… why do we keep it after the process
> > > instance is
> > > > > > > > > finished?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Feb 19, 2024 at 7:12 AM Francisco Javier
> > > Tirado
> > > > > > > Sarti <
> > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Martin.
> > > > > > > > > > > > > > > After taking a deeper look at this, I realize
> > that
> > > the
> > > > > > > > > behaviour
> > > > > > > > > > is
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > expected one.
> > > > > > > > > > > > > > > Runtimes DB does not track the completed
> process
> > > > > instance
> > > > > > > > > (that's
> > > > > > > > > > > > what
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > JDBCProcessInstances warn is telling us), but
> > > > > DataIndex,
> > > > > > as
> > > > > > > > > > > expected,
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > tracking it in processes and nodes table. And
> yes
> > > it
> > > > > will
> > > > > > > > grow
> > > > > > > > > > over
> > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > What we need is some configurable purge
> mechanism
> > > for
> > > > > > > > > DataIndex,
> > > > > > > > > > so
> > > > > > > > > > > > it
> > > > > > > > > > > > > > > eventually removes older completed process
> > > instances.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Feb 13, 2024 at 12:59 PM Francisco
> Javier
> > > > > Tirado
> > > > > > > > Sarti
> > > > > > > > > <
> > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Martin,
> > > > > > > > > > > > > > > > Good catch!. Looks like the skipping
> performed
> > > for
> > > > > > > process
> > > > > > > > > > > > instances
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > not applied to node instances. Something we
> > > > > definitely
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > > > > review
> > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > the runtimes side.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Mon, Feb 12, 2024 at 11:59 PM Martin
> Weiler
> > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >> On a somewhat related note, testing a simple
> > > > > workflow
> > > > > > > > (start
> > > > > > > > > > ->
> > > > > > > > > > > > > script
> > > > > > > > > > > > > > > >> node -> end), I see the following messages
> in
> > > the
> > > > > > logs:
> > > > > > > > > > > > > > > >> 2024-02-12 22:49:50,493 28758dde544c WARN
> > > > > > > > > > > > > > > >>
> > > > > > > [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> > > > > > > > > > > > > > > >> (executor-thread-3) Skipping create of
> process
> > > > > > instance
> > > > > > > > id:
> > > > > > > > > > > > > > > >> 7083088e-b899-47cb-b85c-5d9ccb0aa166,
> state: 2
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> So far, so good. And I'd expect to see no
> > trace
> > > of
> > > > > > this
> > > > > > > > > > process
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > >> database if I don't have data audit enabled.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> However, the 'processes' table contains a
> row
> > > with
> > > > > > > > state=2,
> > > > > > > > > > with
> > > > > > > > > > > > > > related
> > > > > > > > > > > > > > > >> entries in the 'nodes' table. In a load
> test,
> > I
> > > see
> > > > > > > these
> > > > > > > > > > tables
> > > > > > > > > > > > > grow
> > > > > > > > > > > > > > > >> significantly over time. Am I missing
> > something
> > > to
> > > > > > have
> > > > > > > > > these
> > > > > > > > > > > > > entries
> > > > > > > > > > > > > > > >> cleaned up automatically?
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> ________________________________________
> > > > > > > > > > > > > > > >> From: Martin Weiler <mweiler@ibm.com.INVALID
> >
> > > > > > > > > > > > > > > >> Sent: Monday, February 12, 2024 3:40 PM
> > > > > > > > > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > > > > > > > > >> Subject: [EXTERNAL] RE: [DISCUSSION]
> > Performance
> > > > > > issues
> > > > > > > > with
> > > > > > > > > > > > > > data-index
> > > > > > > > > > > > > > > >> persistence addon
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Thanks everyone for your input. Based on
> this
> > > > > > > discussion,
> > > > > > > > I
> > > > > > > > > > > opened
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > >> following PR:
> > > > > > > > > > > > > > > >>
> > > > > > > > >
> > https://github.com/apache/incubator-kie-kogito-apps/pull/1985
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> With this change, the performance seems to
> be
> > > stable
> > > > > > > over
> > > > > > > > > > time:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Martin
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> ________________________________________
> > > > > > > > > > > > > > > >> From: Gonzalo Muñoz <gm...@apache.org>
> > > > > > > > > > > > > > > >> Sent: Friday, February 9, 2024 9:42 AM
> > > > > > > > > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > > > > > > > > >> Subject: [EXTERNAL] Re: [DISCUSSION]
> > Performance
> > > > > > issues
> > > > > > > > with
> > > > > > > > > > > > > > data-index
> > > > > > > > > > > > > > > >> persistence addon
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Great work Francisco,
> > > > > > > > > > > > > > > >> Martin, take a look at this link with some
> > > related
> > > > > > tips
> > > > > > > > (in
> > > > > > > > > > case
> > > > > > > > > > > > you
> > > > > > > > > > > > > > > find
> > > > > > > > > > > > > > > >> it useful):
> > > > > > > > > > > > > > > >>
> > > > > > > > > >
> > > https://www.cybertec-postgresql.com/en/index-your-foreign-key/
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> El vie, 9 feb 2024 a las 17:20, Francisco
> > Javier
> > > > > > Tirado
> > > > > > > > > Sarti
> > > > > > > > > > (<
> > > > > > > > > > > > > > > >> ftirados@redhat.com>) escribió:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> > For the moment being, we will keep JPA
> till
> > we
> > > > > > exhaust
> > > > > > > > all
> > > > > > > > > > > > > > > >> possibilities,
> > > > > > > > > > > > > > > >> > let's call switching from jpa to jdbc our
> > > hidden
> > > > > > plan
> > > > > > > B
> > > > > > > > ;)
> > > > > > > > > > > > > > > >> > I already told Martin, but in order
> everyone
> > > to
> > > > > > know,
> > > > > > > > just
> > > > > > > > > > > after
> > > > > > > > > > > > > > > writing
> > > > > > > > > > > > > > > >> > the previous email, I thought "what if
> > > Postgres is
> > > > > > not
> > > > > > > > > > > > > automatically
> > > > > > > > > > > > > > > >> > indexing foreign keys like mysql?" and,
> > eureka
> > > > > > > > > > > > > > > >> > Postgres doc
> > > > > > > > > > > > > > > >> >
> > > > > > > > > >
> > https://www.postgresql.org/docs/current/ddl-constraints.html
> > > > > > > > > > > > > > > >> > Mysql doc
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > >
> > > > > > > > >
> > > > >
> https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> > > > > > > > > > > > > > > >> > These are the relevant excerpt
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > *Postgresql*
> > > > > > > > > > > > > > > >> > *A foreign key must reference columns that
> > > either
> > > > > > are
> > > > > > > a
> > > > > > > > > > > primary
> > > > > > > > > > > > > key
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > >> form
> > > > > > > > > > > > > > > >> > a unique constraint, or are columns from a
> > > > > > non-partial
> > > > > > > > > > unique
> > > > > > > > > > > > > index.
> > > > > > > > > > > > > > > >> This
> > > > > > > > > > > > > > > >> > means that the referenced columns always
> > have
> > > an
> > > > > > index
> > > > > > > > to
> > > > > > > > > > > allow
> > > > > > > > > > > > > > > >> efficient
> > > > > > > > > > > > > > > >> > lookups on whether a referencing row has a
> > > match.
> > > > > > > Since
> > > > > > > > a
> > > > > > > > > > > DELETE
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > >> row
> > > > > > > > > > > > > > > >> > from the referenced table or an UPDATE of
> a
> > > > > > referenced
> > > > > > > > > > column
> > > > > > > > > > > > will
> > > > > > > > > > > > > > > >> require
> > > > > > > > > > > > > > > >> > a scan of the referencing table for rows
> > > matching
> > > > > > the
> > > > > > > > old
> > > > > > > > > > > value,
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > >> > often a good idea to index the referencing
> > > columns
> > > > > > > too.
> > > > > > > > > > > Because
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > >> not
> > > > > > > > > > > > > > > >> > always needed, and there are many choices
> > > > > available
> > > > > > on
> > > > > > > > how
> > > > > > > > > > to
> > > > > > > > > > > > > index,
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > >> > declaration of a foreign key constraint
> does
> > > not
> > > > > > > > > > automatically
> > > > > > > > > > > > > > create
> > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > >> > index on the referencing columns.*
> > > > > > > > > > > > > > > >> > *Mysql*
> > > > > > > > > > > > > > > >> > *MySQL requires that foreign key columns
> be
> > > > > indexed;
> > > > > > > if
> > > > > > > > > you
> > > > > > > > > > > > > create a
> > > > > > > > > > > > > > > >> table
> > > > > > > > > > > > > > > >> > with a foreign key constraint but no index
> > on
> > > a
> > > > > > given
> > > > > > > > > > column,
> > > > > > > > > > > an
> > > > > > > > > > > > > > index
> > > > > > > > > > > > > > > >> is
> > > > > > > > > > > > > > > >> > created. *
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > So I asked Martin to especially create an
> > > index
> > > > > for
> > > > > > > > > > > > > > > process_instance_id
> > > > > > > > > > > > > > > >> > column on nodes table
> > > > > > > > > > > > > > > >> > I think that will fix the problem detected
> > on
> > > the
> > > > > > > thread
> > > > > > > > > > dump.
> > > > > > > > > > > > > > > >> > The simpler process test to verify queries
> > are
> > > > > fine
> > > > > > > > still
> > > > > > > > > > > > stands,
> > > > > > > > > > > > > > > >> though ;)
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > On Fri, Feb 9, 2024 at 5:10 PM Tibor
> > Zimányi <
> > > > > > > > > > > > tzimanyi@apache.org
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > > I always preferred pure JDBC over
> > Hibernate
> > > > > > myself,
> > > > > > > > just
> > > > > > > > > > for
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > sake
> > > > > > > > > > > > > > > >> of
> > > > > > > > > > > > > > > >> > > control of what is happening :) So I
> would
> > > not
> > > > > -1
> > > > > > > that
> > > > > > > > > > > myself.
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> > > Tibor
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> > > Dňa pi 9. 2. 2024, 17:00 Francisco
> Javier
> > > Tirado
> > > > > > > > Sarti <
> > > > > > > > > > > > > > > >> > > ftirados@redhat.com>
> > > > > > > > > > > > > > > >> > > napísal(a):
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> > > > Hi,
> > > > > > > > > > > > > > > >> > > > Usually I do not want to talk about
> work
> > > in
> > > > > > > progress
> > > > > > > > > > > because
> > > > > > > > > > > > > > > >> > preliminary
> > > > > > > > > > > > > > > >> > > > conclusions are pretty volatile but,
> > well,
> > > > > there
> > > > > > > > are a
> > > > > > > > > > > > couple
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > >> things
> > > > > > > > > > > > > > > >> > > > that can be concluded from the really
> > > valuable
> > > > > > > > > > information
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > >> Martin
> > > > > > > > > > > > > > > >> > > > provided.
> > > > > > > > > > > > > > > >> > > > 1) In order to be able to determine if
> > the
> > > > > > number
> > > > > > > of
> > > > > > > > > > > > > statements
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > >> > larger
> > > > > > > > > > > > > > > >> > > > than expected, I asked Martin to test
> > > with a
> > > > > > > simpler
> > > > > > > > > > > process
> > > > > > > > > > > > > > > >> > definition.
> > > > > > > > > > > > > > > >> > > > One with just three nodes: start,
> script
> > > and
> > > > > > end.
> > > > > > > > The
> > > > > > > > > > > script
> > > > > > > > > > > > > one
> > > > > > > > > > > > > > > >> should
> > > > > > > > > > > > > > > >> > > > change just one variable. This way we
> > can
> > > > > > analyze
> > > > > > > if
> > > > > > > > > the
> > > > > > > > > > > > > number
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > >> > > queries
> > > > > > > > > > > > > > > >> > > > is the expected one. From the single
> log
> > > > > (audit
> > > > > > > was
> > > > > > > > > > > > activated
> > > > > > > > > > > > > > > them)
> > > > > > > > > > > > > > > >> my
> > > > > > > > > > > > > > > >> > > > conclusion is that the number of
> > > > > insert/updates
> > > > > > > over
> > > > > > > > > > > > processes
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > >> > nodes
> > > > > > > > > > > > > > > >> > > > (there a lot over task, that I will
> > > prefer to
> > > > > > skip
> > > > > > > > for
> > > > > > > > > > > now,
> > > > > > > > > > > > > baby
> > > > > > > > > > > > > > > >> steps)
> > > > > > > > > > > > > > > >> > > is
> > > > > > > > > > > > > > > >> > > > the expected one.
> > > > > > > > > > > > > > > >> > > > 2) Analysing the thread dump, we see
> > > around 15
> > > > > > > > threads
> > > > > > > > > > > > > executing
> > > > > > > > > > > > > > > >> this
> > > > > > > > > > > > > > > >> > > line
> > > > > > > > > > > > > > > >> > > > at
> > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > > > > > > > > > > > > > >> > > > so its pretty clear the code to be
> > > optimized
> > > > > ;).
> > > > > > > I'm
> > > > > > > > > > > > > evaluating
> > > > > > > > > > > > > > > >> > > > possibilities within JPA/Hibernate,
> but
> > > I'm
> > > > > > > starting
> > > > > > > > > to
> > > > > > > > > > > > think
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > >> it
> > > > > > > > > > > > > > > >> > > might
> > > > > > > > > > > > > > > >> > > > be better to switch to JDBC and skip
> > > > > hibernate.
> > > > > > > Our
> > > > > > > > > > lives
> > > > > > > > > > > > will
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > >> > > simpler,
> > > > > > > > > > > > > > > >> > > > especially with a schema relatively
> > simple
> > > > > like
> > > > > > > ours
> > > > > > > > > > (that
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > >> my
> > > > > > > > > > > > > > > >> > > > recommendation if I was an external
> > > > > consultant)
> > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > >> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor
> > > Zimányi <
> > > > > > > > > > > > > > tzimanyi@apache.org
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > >> > > > > Hi,
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > >> > > > > this will be a bit off-topic.
> However
> > > as far
> > > > > > as
> > > > > > > > > > > > > performance, I
> > > > > > > > > > > > > > > >> think
> > > > > > > > > > > > > > > >> > we
> > > > > > > > > > > > > > > >> > > > > should think about that we have
> string
> > > > > primary
> > > > > > > > keys
> > > > > > > > > > > > (IDs). I
> > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > >> > > expect
> > > > > > > > > > > > > > > >> > > > > the database systems are much better
> > > with
> > > > > > > indexing
> > > > > > > > > > > numeric
> > > > > > > > > > > > > > keys
> > > > > > > > > > > > > > > >> than
> > > > > > > > > > > > > > > >> > > > > strings. I remember from the past,
> > when
> > > I
> > > > > was
> > > > > > > > > working
> > > > > > > > > > > with
> > > > > > > > > > > > > > DBs,
> > > > > > > > > > > > > > > >> that
> > > > > > > > > > > > > > > >> > > > using
> > > > > > > > > > > > > > > >> > > > > strings as keys or indexes was a
> > > discouraged
> > > > > > > > > practice.
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > >> > > > > Best regards,
> > > > > > > > > > > > > > > >> > > > > Tibor
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > >> > > > > Dňa št 8. 2. 2024, 22:45 Martin
> Weiler
> > > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > napísal(a):
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > >> > > > > > I changed the test to use MongoDB
> > [1]
> > > and
> > > > > I
> > > > > > > > don't
> > > > > > > > > > see
> > > > > > > > > > > a
> > > > > > > > > > > > > > > >> performance
> > > > > > > > > > > > > > > >> > > > > > degradation with this setup [2].
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > > > Please keep us posted of your
> > > findings.
> > > > > > > Thanks!
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > > > Martin
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > > > [1]
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > > > > > > > > > > > > >> > > > > > [2]
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > > >
> > > ________________________________________
> > > > > > > > > > > > > > > >> > > > > > From: Francisco Javier Tirado
> Sarti
> > <
> > > > > > > > > > > > ftirados@redhat.com>
> > > > > > > > > > > > > > > >> > > > > > Sent: Wednesday, February 7, 2024
> > > 11:40 AM
> > > > > > > > > > > > > > > >> > > > > > To: dev@kie.apache.org
> > > > > > > > > > > > > > > >> > > > > > Subject: [EXTERNAL] Re:
> [DISCUSSION]
> > > > > > > Performance
> > > > > > > > > > > issues
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > > >> > > data-index
> > > > > > > > > > > > > > > >> > > > > > persistence addon
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > > > yes, it can be index degradation
> > > because
> > > > > of
> > > > > > > > size,
> > > > > > > > > > but
> > > > > > > > > > > I
> > > > > > > > > > > > > > > believe
> > > > > > > > > > > > > > > >> (I
> > > > > > > > > > > > > > > >> > > > might
> > > > > > > > > > > > > > > >> > > > > be
> > > > > > > > > > > > > > > >> > > > > > wrong) the db is too small (yet)
> for
> > > that.
> > > > > > > > > > > > > > > >> > > > > > But, eventually, Postgres, when
> the
> > > DB is
> > > > > > huge
> > > > > > > > > > enough,
> > > > > > > > > > > > > > > >> unavoidably
> > > > > > > > > > > > > > > >> > > will
> > > > > > > > > > > > > > > >> > > > > > behave like the graphic that
> Martin
> > > sent.
> > > > > > > > > > > > > > > >> > > > > > Since I believe we are not huge
> > enough
> > > > > > (yet),
> > > > > > > > lets
> > > > > > > > > > > rule
> > > > > > > > > > > > > out
> > > > > > > > > > > > > > > >> another
> > > > > > > > > > > > > > > >> > > > issue
> > > > > > > > > > > > > > > >> > > > > > by analysing the sql logs (I
> > requested
> > > > > those
> > > > > > > to
> > > > > > > > > > Martin
> > > > > > > > > > > > > > offline
> > > > > > > > > > > > > > > >> and
> > > > > > > > > > > > > > > >> > he
> > > > > > > > > > > > > > > >> > > > is
> > > > > > > > > > > > > > > >> > > > > > going to kindly collect them).
> > > > > > > > > > > > > > > >> > > > > > Also Im curious to know if Mongo
> > > behave in
> > > > > > the
> > > > > > > > > same
> > > > > > > > > > > way.
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > > > On Wed, Feb 7, 2024 at 7:25 PM
> > Enrique
> > > > > > > Gonzalez
> > > > > > > > > > > > Martinez <
> > > > > > > > > > > > > > > >> > > > > > egonzalez@apache.org> wrote:
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > > > > Hi Francisco,
> > > > > > > > > > > > > > > >> > > > > > > I would highly recommend to
> check
> > > > > indexes
> > > > > > > and
> > > > > > > > > how
> > > > > > > > > > > the
> > > > > > > > > > > > > > > updates
> > > > > > > > > > > > > > > >> > work
> > > > > > > > > > > > > > > >> > > in
> > > > > > > > > > > > > > > >> > > > > > data
> > > > > > > > > > > > > > > >> > > > > > > index to avoid full scan table
> and
> > > lock
> > > > > > the
> > > > > > > > full
> > > > > > > > > > > > table.
> > > > > > > > > > > > > > Some
> > > > > > > > > > > > > > > >> db
> > > > > > > > > > > > > > > >> > are
> > > > > > > > > > > > > > > >> > > > > very
> > > > > > > > > > > > > > > >> > > > > > > sensitive to that.
> > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > El mié, 7 feb 2024, 18:41,
> > Francisco
> > > > > > Javier
> > > > > > > > > Tirado
> > > > > > > > > > > > > Sarti <
> > > > > > > > > > > > > > > >> > > > > > > ftirados@redhat.com> escribió:
> > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > Hi Martin,
> > > > > > > > > > > > > > > >> > > > > > > > While I analyze the data, let
> me
> > > ask
> > > > > you
> > > > > > > if
> > > > > > > > it
> > > > > > > > > > is
> > > > > > > > > > > > > > possible
> > > > > > > > > > > > > > > >> to
> > > > > > > > > > > > > > > >> > > > perform
> > > > > > > > > > > > > > > >> > > > > > > > another check (similar in a
> way
> > to
> > > > > > > disabling
> > > > > > > > > > > > > data-index
> > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > >> > you
> > > > > > > > > > > > > > > >> > > > do)
> > > > > > > > > > > > > > > >> > > > > > Can
> > > > > > > > > > > > > > > >> > > > > > > > you switch to MongoDB
> > persistence
> > > and
> > > > > > > check
> > > > > > > > if
> > > > > > > > > > the
> > > > > > > > > > > > > same
> > > > > > > > > > > > > > > >> > > degradation
> > > > > > > > > > > > > > > >> > > > > > that
> > > > > > > > > > > > > > > >> > > > > > > is
> > > > > > > > > > > > > > > >> > > > > > > > there for postgres remains?
> > > > > > > > > > > > > > > >> > > > > > > > I do not know if this is
> > feasible
> > > but
> > > > > > will
> > > > > > > > > > > certainly
> > > > > > > > > > > > > > > >> indicate
> > > > > > > > > > > > > > > >> > the
> > > > > > > > > > > > > > > >> > > > > > problem
> > > > > > > > > > > > > > > >> > > > > > > > is on the postgres storage
> layer
> > > and I
> > > > > > do
> > > > > > > > not
> > > > > > > > > > > have a
> > > > > > > > > > > > > > clear
> > > > > > > > > > > > > > > >> > > > prediction
> > > > > > > > > > > > > > > >> > > > > > of
> > > > > > > > > > > > > > > >> > > > > > > > what we will see when doing
> this
> > > > > switch.
> > > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM
> > > Martin
> > > > > > > Weiler
> > > > > > > > > > > > > > > >> > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > wrote:
> > > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > Hi Francisco,
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > thanks for your work on this
> > > > > important
> > > > > > > > > topic!
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > I would like to share some
> > test
> > > > > > results
> > > > > > > > > here,
> > > > > > > > > > > > which
> > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > >> > help
> > > > > > > > > > > > > > > >> > > to
> > > > > > > > > > > > > > > >> > > > > > > improve
> > > > > > > > > > > > > > > >> > > > > > > > > the codebase even further. I
> > am
> > > > > using
> > > > > > > the
> > > > > > > > > > jmeter
> > > > > > > > > > > > > based
> > > > > > > > > > > > > > > >> test
> > > > > > > > > > > > > > > >> > > case
> > > > > > > > > > > > > > > >> > > > > from
> > > > > > > > > > > > > > > >> > > > > > > > Pere
> > > > > > > > > > > > > > > >> > > > > > > > > and Enrique (thanks guys!)
> [1]
> > > which
> > > > > > > uses
> > > > > > > > a
> > > > > > > > > > load
> > > > > > > > > > > > of
> > > > > > > > > > > > > 30
> > > > > > > > > > > > > > > >> > threads
> > > > > > > > > > > > > > > >> > > to
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > 1) start a new process
> > instance
> > > > > (POST)
> > > > > > > > > > > > > > > >> > > > > > > > > 2) retrieve tasks for a user
> > > (GET)
> > > > > > > > > > > > > > > >> > > > > > > > > 3) fetches task details
> (GET)
> > > > > > > > > > > > > > > >> > > > > > > > > 4) complete a task (POST)
> > > > > > > > > > > > > > > >> > > > > > > > > 5) execute a query on
> > data-audit
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > With this test setup, I
> > noticed
> > > that
> > > > > > the
> > > > > > > > > > > > performance
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > > >> > > POST
> > > > > > > > > > > > > > > >> > > > > > > > > requests, in particular the
> > one
> > > to
> > > > > > > start a
> > > > > > > > > new
> > > > > > > > > > > > > process
> > > > > > > > > > > > > > > >> > > instance,
> > > > > > > > > > > > > > > >> > > > > > > degrades
> > > > > > > > > > > > > > > >> > > > > > > > > over time - see graph [2].
> If
> > I
> > > run
> > > > > > the
> > > > > > > > same
> > > > > > > > > > > test
> > > > > > > > > > > > > > > without
> > > > > > > > > > > > > > > >> > > > > data-index,
> > > > > > > > > > > > > > > >> > > > > > > > then
> > > > > > > > > > > > > > > >> > > > > > > > > there is no such performance
> > > > > > degradation
> > > > > > > > > [3].
> > > > > > > > > > > You
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > >> find a
> > > > > > > > > > > > > > > >> > > > thread
> > > > > > > > > > > > > > > >> > > > > > > dump
> > > > > > > > > > > > > > > >> > > > > > > > > captured a few minutes into
> > the
> > > > > first
> > > > > > > test
> > > > > > > > > > here
> > > > > > > > > > > > [4]
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > >> > might
> > > > > > > > > > > > > > > >> > > > help
> > > > > > > > > > > > > > > >> > > > > > to
> > > > > > > > > > > > > > > >> > > > > > > > see
> > > > > > > > > > > > > > > >> > > > > > > > > some of the contention
> points.
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > I'd appreciate if you could
> > > take a
> > > > > > look
> > > > > > > > and
> > > > > > > > > > see
> > > > > > > > > > > if
> > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > >> is
> > > > > > > > > > > > > > > >> > > > > something
> > > > > > > > > > > > > > > >> > > > > > > > that
> > > > > > > > > > > > > > > >> > > > > > > > > can be further improved
> based
> > on
> > > > > your
> > > > > > > > > previous
> > > > > > > > > > > > work.
> > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > >> you
> > > > > > > > > > > > > > > >> > > need
> > > > > > > > > > > > > > > >> > > > > any
> > > > > > > > > > > > > > > >> > > > > > > > > additional data, let me
> know,
> > > but
> > > > > > > > otherwise
> > > > > > > > > it
> > > > > > > > > > > is
> > > > > > > > > > > > > > > >> > > straightforward
> > > > > > > > > > > > > > > >> > > > > to
> > > > > > > > > > > > > > > >> > > > > > > run
> > > > > > > > > > > > > > > >> > > > > > > > > the jmeter test as well.
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > Thanks,
> > > > > > > > > > > > > > > >> > > > > > > > > Martin
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > [1]
> > > > > > > > > > > > > > > >>
> > > > > > https://github.com/pefernan/job-service-refactor-test/
> > > > > > > > > > > > > > > >> > > > > > > > > [2]
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > > > > > > > > > > > >> > > > > > > > > [3]
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > > > > > > > > > > > >> > > > > > > > > [4]
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > ________________________________________
> > > > > > > > > > > > > > > >> > > > > > > > > From: Francisco Javier
> Tirado
> > > Sarti
> > > > > <
> > > > > > > > > > > > > > > ftirados@redhat.com>
> > > > > > > > > > > > > > > >> > > > > > > > > Sent: Wednesday, January 17,
> > > 2024
> > > > > 9:13
> > > > > > > AM
> > > > > > > > > > > > > > > >> > > > > > > > > To: dev@kie.apache.org
> > > > > > > > > > > > > > > >> > > > > > > > > Cc: Pere Fernandez Perez
> > > > > > > > > > > > > > > >> > > > > > > > > Subject: [EXTERNAL] Re:
> > > [DISCUSSION]
> > > > > > > > > > Performance
> > > > > > > > > > > > > > issues
> > > > > > > > > > > > > > > >> with
> > > > > > > > > > > > > > > >> > > > > > data-index
> > > > > > > > > > > > > > > >> > > > > > > > > persistence addon
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > Hi Alex,
> > > > > > > > > > > > > > > >> > > > > > > > > I did not take times (which
> > > depends
> > > > > > on a
> > > > > > > > > > number
> > > > > > > > > > > of
> > > > > > > > > > > > > > > >> variables
> > > > > > > > > > > > > > > >> > > that
> > > > > > > > > > > > > > > >> > > > > > > > > drastically change between
> > > > > > > environments),
> > > > > > > > > but
> > > > > > > > > > > > verify
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > > >> > > > > number
> > > > > > > > > > > > > > > >> > > > > > of
> > > > > > > > > > > > > > > >> > > > > > > > > updates has been reduced
> > > drastically
> > > > > > > > without
> > > > > > > > > > > > losing
> > > > > > > > > > > > > > > >> > > > functionality,
> > > > > > > > > > > > > > > >> > > > > > > which
> > > > > > > > > > > > > > > >> > > > > > > > is
> > > > > > > > > > > > > > > >> > > > > > > > > objectively a good thing. If
> > > before
> > > > > > the
> > > > > > > > > > change,
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > every
> > > > > > > > > > > > > > > >> > node
> > > > > > > > > > > > > > > >> > > > > > > executed,
> > > > > > > > > > > > > > > >> > > > > > > > we
> > > > > > > > > > > > > > > >> > > > > > > > > have an update for every
> node
> > > > > > previously
> > > > > > > > > > > executed,
> > > > > > > > > > > > > so
> > > > > > > > > > > > > > > if a
> > > > > > > > > > > > > > > >> > > > process
> > > > > > > > > > > > > > > >> > > > > > have
> > > > > > > > > > > > > > > >> > > > > > > > 50
> > > > > > > > > > > > > > > >> > > > > > > > > nodes to execute, we were
> > > performing
> > > > > > > > nearly
> > > > > > > > > > > > 50*51/2
> > > > > > > > > > > > > > > >> updates,
> > > > > > > > > > > > > > > >> > > > which
> > > > > > > > > > > > > > > >> > > > > > > gives
> > > > > > > > > > > > > > > >> > > > > > > > us
> > > > > > > > > > > > > > > >> > > > > > > > > a total of  1275 updates,
> now
> > we
> > > > > have
> > > > > > > just
> > > > > > > > > one
> > > > > > > > > > > for
> > > > > > > > > > > > > > every
> > > > > > > > > > > > > > > >> node
> > > > > > > > > > > > > > > >> > > > being
> > > > > > > > > > > > > > > >> > > > > > > > > executed, implying a total
> of
> > 50
>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Francisco Javier Tirado Sarti <ft...@redhat.com>]

Ok, so we have two flavours, clean up on demand through REST user call
(that's fine, the only issue is the scope of it. For example, should we
allow a bulk delete api where you specify the process instances, if the
user knows them somehow, that user wants to delete or just, as we are
proposing, delete all instances that have been there for a while?) and cron
process (that is a replacement of the properties approach I recommend).
In my opinion, if we have the explicit REST API, we should not use the
cron.

On Thu, Feb 22, 2024 at 6:32 AM Enrique Gonzalez Martinez <
egonzalez@apache.org> wrote:

> +1 alex
>
> El jue, 22 feb 2024, 0:14, Alex Porcelli <al...@porcelli.me> escribió:
>
> > I think I need to clarify what I mean by REST...
> >
> > The REST endpoint I have in mind would execute the necessary cleaning
> > based on an input parameter (ie. instanced closed for more than X days
> > or something like that).
> >
> > The use of cron is that an external cron could call the cleaning REST
> > endpoint from time to time (once a month, on every sunday night, etc).
> >
> >
> > On Wed, Feb 21, 2024 at 10:50 AM Francisco Javier Tirado Sarti
> > <ft...@redhat.com> wrote:
> > >
> > > Hi Alex,
> > > I'm wondering how cron and rest can live together.
> > > With cron the responsibility to do the clean up relies on a process
> > > external to the microservice accessing the db to be purged, while where
> > > using REST API the process performing the clean up is typically (but
> not
> > > necessarily) the same microservice that accesses the DB. In that sense,
> > > REST API is similar to using properties (in both approaches the
> > responsible
> > > to clean up the data is the microservice), but they are different
> because
> > > REST is more sophisticated (but compatible, implementation wise we can
> > use
> > > a ConfigSource, that is updated by the REST and read, as a property, by
> > the
> > > cleanup routine). So, probably I'm missing something but I see the cron
> > > approach not reconcilable with the REST approach.
> > > Anyway just a detail, I agree the purge approach should be similar for
> > all
> > > microservices
> > >
> > > On Wed, Feb 21, 2024 at 4:37 PM Alex Porcelli <po...@apache.org>
> > wrote:
> > >
> > > > +1 for provide consistent behavior across the board
> > > >
> > > > my 2c:
> > > >
> > > > +1 for REST interface (preferred for external general management
> > > > perspective)
> > > > -1 for properties to auto clean; this can be achieved with cron and
> > rest
> > > > call
> > > >
> > > > Now about GraphQL: the platform today is heavily invested in such
> > > > interface, so it might make sense. BUT I’d consider it in a second
> > moment.
> > > >
> > > >
> > > > On Wed, Feb 21, 2024 at 6:52 AM Francisco Javier Tirado Sarti <
> > > > ftirados@redhat.com> wrote:
> > > >
> > > > > Hi Enrique,
> > > > > If we configure such policies through properties, it won't be
> enough
> > to
> > > > > define a naming convention and rely on the target platform
> > configuration
> > > > > capabilities (SpringBoot or Quarkus)?.
> > > > >
> > > > > On Wed, Feb 21, 2024 at 12:36 PM Enrique Gonzalez Martinez <
> > > > > egonzalez@apache.org> wrote:
> > > > >
> > > > > > Hi Francisco
> > > > > >
> > > > > > The discussion we need to have before how to achieve certain
> > features,
> > > > is
> > > > > > the overall of the user experience. If you want to clean it up
> > that way
> > > > > it
> > > > > > is fine by me. I have nothing against setting policy related to
> > some
> > > > sort
> > > > > > of clean up for a process completed some time ago.
> > > > > >
> > > > > > As we discussed previously the data index is a snapshot of the
> last
> > > > state
> > > > > > of the process instance included completed, but nothing was said
> > once
> > > > > > completed when we need to clean that up so any policy is welcome.
> > How
> > > > to
> > > > > > achieve that policy is something completely different.
> > > > > >
> > > > > > The main problem is how every microservice is exposing that API
> to
> > the
> > > > > end
> > > > > > user or consumer (other system) and which system is going to be
> the
> > > > > façade
> > > > > > for complex deployments and operations. That is the discussion I
> > was
> > > > > > mentioning before.
> > > > > >
> > > > > > What I want to avoid is to set different policies among
> components
> > and
> > > > we
> > > > > > should strive to be as much as possible to offer certain
> > capabilities
> > > > in
> > > > > > the same fashion, e.g: clean up mechanism.
> > > > > > In the same way I want to avoid making the current system more
> > complex
> > > > > than
> > > > > > they are. So far they are aligned offering one simple
> > responsibility
> > > > but
> > > > > I
> > > > > > would be against mixing for instance job service with data index.
> > > > > >
> > > > > > El mié, 21 feb 2024 a las 12:13, Francisco Javier Tirado Sarti (<
> > > > > > ftirados@redhat.com>) escribió:
> > > > > >
> > > > > > > Hi Enrique,
> > > > > > > In the case of data index I think the data to be purged is
> > finished
> > > > > > process
> > > > > > > instances (I do not think we should remove process instance
> that
> > has
> > > > > been
> > > > > > > alive for ages, even if it is very likely they are not going to
> > be
> > > > ever
> > > > > > > completed)
> > > > > > > Once you delete those process instances, you also delete the
> > > > associated
> > > > > > > user tasks and jobs.
> > > > > > > Therefore the problem is relatively simple, to be able to
> > configure
> > > > how
> > > > > > > much time a completed process instance should remain in the
> data
> > > > index
> > > > > > > database. We can take a simple approach: a property with a min
> > > > duration
> > > > > > > that cannot be changed once data index is started; a slightly
> > complex
> > > > > > one:
> > > > > > > the same property but watching it to react for changes; or the
> > full
> > > > > > suite:
> > > > > > > an admin API to be able to change the policy at any moment.
> > > > > > > I think this discussion is worth having.
> > > > > > >
> > > > > > > On Wed, Feb 21, 2024 at 6:15 AM Enrique Gonzalez Martinez <
> > > > > > > egonzalez@apache.org> wrote:
> > > > > > >
> > > > > > > > Hi Martin, the main problem regarding the purge is because it
> > is
> > > > > still
> > > > > > > > unclear the policy for what tech to use and the future
> > components.
> > > > > > > >
> > > > > > > > Recently we had a discussion about proposing graphql for this
> > sort
> > > > of
> > > > > > > admin
> > > > > > > > tasks. So far for subsystems we have been using rest
> endpoints
> > > > (like
> > > > > > > update
> > > > > > > > timers or modify human task or change processes). There is
> one
> > > > > > exception
> > > > > > > > which is the gateway that is pure graphql and somehow uses
> > graphql
> > > > > for
> > > > > > > > everything making complex operations under the hood.
> > > > > > > >
> > > > > > > > This has somehow frozen the purge for data audit for a bit
> and
> > the
> > > > > > > proposal
> > > > > > > > was to use rest endpoints to do the clean up in the component
> > and
> > > > > offer
> > > > > > > the
> > > > > > > > graphql counterpart in the gateway promoting it to a first
> > class
> > > > > > citizen
> > > > > > > > component instead of having it embedded in the data index.
> > > > > > > >
> > > > > > > > I would suggest to come up at least with a policy first
> > regarding
> > > > the
> > > > > > > > convention every component should address this.
> > > > > > > >
> > > > > > > >
> > > > > > > > El mar, 20 feb 2024, 23:39, Martin Weiler
> > <mweiler@ibm.com.invalid
> > > > >
> > > > > > > > escribió:
> > > > > > > >
> > > > > > > > > IMO, it is good to have this discussion around data sanity
> > now
> > > > > > instead
> > > > > > > of
> > > > > > > > > putting it off until later when data has already
> accumulated
> > in
> > > > > > > > production
> > > > > > > > > environments.
> > > > > > > > >
> > > > > > > > > Based on the input here, we are dealing with three types of
> > data:
> > > > > > > > > 1. Runtime data - active instances only, engine cleans up
> the
> > > > data
> > > > > > > > > automatically at process instance end
> > > > > > > > > 2. Historic log data - data created by data-audit intended
> > for
> > > > long
> > > > > > > term
> > > > > > > > > storage
> > > > > > > > > 3. Data-index data - somehow this data falls in between the
> > two
> > > > > > > > > aforementioned categories, with the idea of the data being
> > > > > "recent",
> > > > > > > but
> > > > > > > > > not restricted to active instances only
> > > > > > > > >
> > > > > > > > > We'd need purge strategies for both #2 and #3 (perhaps
> > different
> > > > > > ones,
> > > > > > > or
> > > > > > > > > with different config settings) in order to prevent
> unlimited
> > > > data
> > > > > > > > growth.
> > > > > > > > >
> > > > > > > > > ________________________________________
> > > > > > > > > From: Enrique Gonzalez Martinez <eg...@apache.org>
> > > > > > > > > Sent: Monday, February 19, 2024 7:11 AM
> > > > > > > > > To: dev@kie.apache.org
> > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues
> with
> > > > > > data-index
> > > > > > > > > persistence addon
> > > > > > > > >
> > > > > > > > > Hi Francisco,
> > > > > > > > > To give you more context about this.
> > > > > > > > >
> > > > > > > > > STP is a concept, a process with certain constraints: no
> > > > > persistence
> > > > > > > and
> > > > > > > > > returning the outcome in the call (sync execution with no
> > idle
> > > > > > states).
> > > > > > > > It
> > > > > > > > > was a requirement from a user in the past. One of the
> > > > requirements
> > > > > > was
> > > > > > > > > leaving no trail. In v7 was easy because you could disable
> > the
> > > > > audit
> > > > > > in
> > > > > > > > > that case. Actually we have the same way to do what we did
> > in v7
> > > > in
> > > > > > > here
> > > > > > > > as
> > > > > > > > > you can add/remove index just removing deps.
> > > > > > > > >
> > > > > > > > > We have the same outcome with different approaches and STP
> is
> > > > > already
> > > > > > > > > delivered.
> > > > > > > > >
> > > > > > > > > El lun, 19 feb 2024 a las 14:46, Francisco Javier Tirado
> > Sarti (<
> > > > > > > > > ftirados@redhat.com>) escribió:
> > > > > > > > >
> > > > > > > > > > Regarding STP (which is not a concept that we have in the
> > > > code. I
> > > > > > > mean
> > > > > > > > > STP
> > > > > > > > > > are processes as nonSTP are), I guess, as all processes,
> > they
> > > > > were
> > > > > > > kept
> > > > > > > > > in
> > > > > > > > > > DataIndex once completed because users wanted (and still
> > wants)
> > > > > to
> > > > > > > > check
> > > > > > > > > > the result once the call had been performed. If we want
> to
> > > > leave
> > > > > no
> > > > > > > > trace
> > > > > > > > > > of them in DataIndex for some reason, we will need to
> make
> > it a
> > > > > > > > > > Runtimes concept so DataIndex can handle them in a
> > different
> > > > way.
> > > > > > > > > >
> > > > > > > > > > On Mon, Feb 19, 2024 at 2:27 PM Enrique Gonzalez
> Martinez <
> > > > > > > > > > egonzalez@apache.org> wrote:
> > > > > > > > > >
> > > > > > > > > > > Alex:
> > > > > > > > > > > Right now the data index is working in the same way as
> > it did
> > > > > in
> > > > > > v7
> > > > > > > > > with
> > > > > > > > > > > the emitters. The only difference between two impl is
> > that in
> > > > > > here
> > > > > > > > the
> > > > > > > > > > > storage is pgsql instead elastic search.  You are right
> > > > > regarding
> > > > > > > is
> > > > > > > > a
> > > > > > > > > > > snapshot of the last state of the process but we did
> > never
> > > > > define
> > > > > > > how
> > > > > > > > > > long
> > > > > > > > > > > would be alive that dats Honestly i am happy right now
> > with
> > > > the
> > > > > > way
> > > > > > > > it
> > > > > > > > > > > works. The clean up mechanism is still tbd because we
> > still
> > > > > need
> > > > > > to
> > > > > > > > > > discuss
> > > > > > > > > > > other stuff first.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Regarding stp is to leave no trail because u can get
> the
> > > > > outcome
> > > > > > > > > directly
> > > > > > > > > > > from the call. It was defined like that in v7. So there
> > is no
> > > > > use
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > > index or the audit.
> > > > > > > > > > >
> > > > > > > > > > > El lun, 19 feb 2024, 14:13, Francisco Javier Tirado
> > Sarti <
> > > > > > > > > > > ftirados@redhat.com> escribió:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > > There has been some confusion about the purpose of
> > > > DataIndex.
> > > > > > To
> > > > > > > be
> > > > > > > > > > > honest
> > > > > > > > > > > > I believe they were already sorted out, but your
> e-mail
> > > > makes
> > > > > > me
> > > > > > > > > think
> > > > > > > > > > > that
> > > > > > > > > > > > is not the case ;). I let Kris to clarify that with
> > you. My
> > > > > > view
> > > > > > > is
> > > > > > > > > > that
> > > > > > > > > > > > data-index is a way to query recently closed and
> active
> > > > > > processes
> > > > > > > > > (the
> > > > > > > > > > > key
> > > > > > > > > > > > here is the definition of recently, which in my
> opinion
> > > > > should
> > > > > > be
> > > > > > > > > > > > configurable)
> > > > > > > > > > > > But, besides that discussion and being pragmatic,
> > keeping
> > > > > > > finishing
> > > > > > > > > > > process
> > > > > > > > > > > > instances "for a while" in DataIndex was the only way
> > for
> > > > > users
> > > > > > > to
> > > > > > > > > > query
> > > > > > > > > > > > the result of straight through processes. That's a
> > function
> > > > > > that
> > > > > > > > > cannot
> > > > > > > > > > > be
> > > > > > > > > > > > removed right now
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Feb 19, 2024 at 1:33 PM Alex Porcelli <
> > > > > > > porcelli@apache.org
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > if data index was supposed to provide snapshot view
> > of
> > > > the
> > > > > > > > process
> > > > > > > > > > > > > instance… why do we keep it after the process
> > instance is
> > > > > > > > finished?
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Feb 19, 2024 at 7:12 AM Francisco Javier
> > Tirado
> > > > > > Sarti <
> > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Martin.
> > > > > > > > > > > > > > After taking a deeper look at this, I realize
> that
> > the
> > > > > > > > behaviour
> > > > > > > > > is
> > > > > > > > > > > the
> > > > > > > > > > > > > > expected one.
> > > > > > > > > > > > > > Runtimes DB does not track the completed process
> > > > instance
> > > > > > > > (that's
> > > > > > > > > > > what
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > JDBCProcessInstances warn is telling us), but
> > > > DataIndex,
> > > > > as
> > > > > > > > > > expected,
> > > > > > > > > > > > is
> > > > > > > > > > > > > > tracking it in processes and nodes table. And yes
> > it
> > > > will
> > > > > > > grow
> > > > > > > > > over
> > > > > > > > > > > > time.
> > > > > > > > > > > > > > What we need is some configurable purge mechanism
> > for
> > > > > > > > DataIndex,
> > > > > > > > > so
> > > > > > > > > > > it
> > > > > > > > > > > > > > eventually removes older completed process
> > instances.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Feb 13, 2024 at 12:59 PM Francisco Javier
> > > > Tirado
> > > > > > > Sarti
> > > > > > > > <
> > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Martin,
> > > > > > > > > > > > > > > Good catch!. Looks like the skipping performed
> > for
> > > > > > process
> > > > > > > > > > > instances
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > not applied to node instances. Something we
> > > > definitely
> > > > > > need
> > > > > > > > to
> > > > > > > > > > > review
> > > > > > > > > > > > > on
> > > > > > > > > > > > > > > the runtimes side.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, Feb 12, 2024 at 11:59 PM Martin Weiler
> > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> On a somewhat related note, testing a simple
> > > > workflow
> > > > > > > (start
> > > > > > > > > ->
> > > > > > > > > > > > script
> > > > > > > > > > > > > > >> node -> end), I see the following messages in
> > the
> > > > > logs:
> > > > > > > > > > > > > > >> 2024-02-12 22:49:50,493 28758dde544c WARN
> > > > > > > > > > > > > > >>
> > > > > > [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> > > > > > > > > > > > > > >> (executor-thread-3) Skipping create of process
> > > > > instance
> > > > > > > id:
> > > > > > > > > > > > > > >> 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> So far, so good. And I'd expect to see no
> trace
> > of
> > > > > this
> > > > > > > > > process
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > >> database if I don't have data audit enabled.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> However, the 'processes' table contains a row
> > with
> > > > > > > state=2,
> > > > > > > > > with
> > > > > > > > > > > > > related
> > > > > > > > > > > > > > >> entries in the 'nodes' table. In a load test,
> I
> > see
> > > > > > these
> > > > > > > > > tables
> > > > > > > > > > > > grow
> > > > > > > > > > > > > > >> significantly over time. Am I missing
> something
> > to
> > > > > have
> > > > > > > > these
> > > > > > > > > > > > entries
> > > > > > > > > > > > > > >> cleaned up automatically?
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> ________________________________________
> > > > > > > > > > > > > > >> From: Martin Weiler <mw...@ibm.com.INVALID>
> > > > > > > > > > > > > > >> Sent: Monday, February 12, 2024 3:40 PM
> > > > > > > > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > > > > > > > >> Subject: [EXTERNAL] RE: [DISCUSSION]
> Performance
> > > > > issues
> > > > > > > with
> > > > > > > > > > > > > data-index
> > > > > > > > > > > > > > >> persistence addon
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Thanks everyone for your input. Based on this
> > > > > > discussion,
> > > > > > > I
> > > > > > > > > > opened
> > > > > > > > > > > > the
> > > > > > > > > > > > > > >> following PR:
> > > > > > > > > > > > > > >>
> > > > > > > >
> https://github.com/apache/incubator-kie-kogito-apps/pull/1985
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> With this change, the performance seems to be
> > stable
> > > > > > over
> > > > > > > > > time:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Martin
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> ________________________________________
> > > > > > > > > > > > > > >> From: Gonzalo Muñoz <gm...@apache.org>
> > > > > > > > > > > > > > >> Sent: Friday, February 9, 2024 9:42 AM
> > > > > > > > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > > > > > > > >> Subject: [EXTERNAL] Re: [DISCUSSION]
> Performance
> > > > > issues
> > > > > > > with
> > > > > > > > > > > > > data-index
> > > > > > > > > > > > > > >> persistence addon
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Great work Francisco,
> > > > > > > > > > > > > > >> Martin, take a look at this link with some
> > related
> > > > > tips
> > > > > > > (in
> > > > > > > > > case
> > > > > > > > > > > you
> > > > > > > > > > > > > > find
> > > > > > > > > > > > > > >> it useful):
> > > > > > > > > > > > > > >>
> > > > > > > > >
> > https://www.cybertec-postgresql.com/en/index-your-foreign-key/
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> El vie, 9 feb 2024 a las 17:20, Francisco
> Javier
> > > > > Tirado
> > > > > > > > Sarti
> > > > > > > > > (<
> > > > > > > > > > > > > > >> ftirados@redhat.com>) escribió:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> > For the moment being, we will keep JPA till
> we
> > > > > exhaust
> > > > > > > all
> > > > > > > > > > > > > > >> possibilities,
> > > > > > > > > > > > > > >> > let's call switching from jpa to jdbc our
> > hidden
> > > > > plan
> > > > > > B
> > > > > > > ;)
> > > > > > > > > > > > > > >> > I already told Martin, but in order everyone
> > to
> > > > > know,
> > > > > > > just
> > > > > > > > > > after
> > > > > > > > > > > > > > writing
> > > > > > > > > > > > > > >> > the previous email, I thought "what if
> > Postgres is
> > > > > not
> > > > > > > > > > > > automatically
> > > > > > > > > > > > > > >> > indexing foreign keys like mysql?" and,
> eureka
> > > > > > > > > > > > > > >> > Postgres doc
> > > > > > > > > > > > > > >> >
> > > > > > > > >
> https://www.postgresql.org/docs/current/ddl-constraints.html
> > > > > > > > > > > > > > >> > Mysql doc
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > >
> > > > > > > >
> > > > https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> > > > > > > > > > > > > > >> > These are the relevant excerpt
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > *Postgresql*
> > > > > > > > > > > > > > >> > *A foreign key must reference columns that
> > either
> > > > > are
> > > > > > a
> > > > > > > > > > primary
> > > > > > > > > > > > key
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > >> form
> > > > > > > > > > > > > > >> > a unique constraint, or are columns from a
> > > > > non-partial
> > > > > > > > > unique
> > > > > > > > > > > > index.
> > > > > > > > > > > > > > >> This
> > > > > > > > > > > > > > >> > means that the referenced columns always
> have
> > an
> > > > > index
> > > > > > > to
> > > > > > > > > > allow
> > > > > > > > > > > > > > >> efficient
> > > > > > > > > > > > > > >> > lookups on whether a referencing row has a
> > match.
> > > > > > Since
> > > > > > > a
> > > > > > > > > > DELETE
> > > > > > > > > > > > of
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > >> row
> > > > > > > > > > > > > > >> > from the referenced table or an UPDATE of a
> > > > > referenced
> > > > > > > > > column
> > > > > > > > > > > will
> > > > > > > > > > > > > > >> require
> > > > > > > > > > > > > > >> > a scan of the referencing table for rows
> > matching
> > > > > the
> > > > > > > old
> > > > > > > > > > value,
> > > > > > > > > > > > it
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > >> > often a good idea to index the referencing
> > columns
> > > > > > too.
> > > > > > > > > > Because
> > > > > > > > > > > > this
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > >> not
> > > > > > > > > > > > > > >> > always needed, and there are many choices
> > > > available
> > > > > on
> > > > > > > how
> > > > > > > > > to
> > > > > > > > > > > > index,
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > >> > declaration of a foreign key constraint does
> > not
> > > > > > > > > automatically
> > > > > > > > > > > > > create
> > > > > > > > > > > > > > an
> > > > > > > > > > > > > > >> > index on the referencing columns.*
> > > > > > > > > > > > > > >> > *Mysql*
> > > > > > > > > > > > > > >> > *MySQL requires that foreign key columns be
> > > > indexed;
> > > > > > if
> > > > > > > > you
> > > > > > > > > > > > create a
> > > > > > > > > > > > > > >> table
> > > > > > > > > > > > > > >> > with a foreign key constraint but no index
> on
> > a
> > > > > given
> > > > > > > > > column,
> > > > > > > > > > an
> > > > > > > > > > > > > index
> > > > > > > > > > > > > > >> is
> > > > > > > > > > > > > > >> > created. *
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > So I asked Martin to especially create an
> > index
> > > > for
> > > > > > > > > > > > > > process_instance_id
> > > > > > > > > > > > > > >> > column on nodes table
> > > > > > > > > > > > > > >> > I think that will fix the problem detected
> on
> > the
> > > > > > thread
> > > > > > > > > dump.
> > > > > > > > > > > > > > >> > The simpler process test to verify queries
> are
> > > > fine
> > > > > > > still
> > > > > > > > > > > stands,
> > > > > > > > > > > > > > >> though ;)
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > On Fri, Feb 9, 2024 at 5:10 PM Tibor
> Zimányi <
> > > > > > > > > > > tzimanyi@apache.org
> > > > > > > > > > > > >
> > > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > > I always preferred pure JDBC over
> Hibernate
> > > > > myself,
> > > > > > > just
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > > > sake
> > > > > > > > > > > > > > >> of
> > > > > > > > > > > > > > >> > > control of what is happening :) So I would
> > not
> > > > -1
> > > > > > that
> > > > > > > > > > myself.
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > > Tibor
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > > Dňa pi 9. 2. 2024, 17:00 Francisco Javier
> > Tirado
> > > > > > > Sarti <
> > > > > > > > > > > > > > >> > > ftirados@redhat.com>
> > > > > > > > > > > > > > >> > > napísal(a):
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > > > Hi,
> > > > > > > > > > > > > > >> > > > Usually I do not want to talk about work
> > in
> > > > > > progress
> > > > > > > > > > because
> > > > > > > > > > > > > > >> > preliminary
> > > > > > > > > > > > > > >> > > > conclusions are pretty volatile but,
> well,
> > > > there
> > > > > > > are a
> > > > > > > > > > > couple
> > > > > > > > > > > > of
> > > > > > > > > > > > > > >> things
> > > > > > > > > > > > > > >> > > > that can be concluded from the really
> > valuable
> > > > > > > > > information
> > > > > > > > > > > > that
> > > > > > > > > > > > > > >> Martin
> > > > > > > > > > > > > > >> > > > provided.
> > > > > > > > > > > > > > >> > > > 1) In order to be able to determine if
> the
> > > > > number
> > > > > > of
> > > > > > > > > > > > statements
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > >> > larger
> > > > > > > > > > > > > > >> > > > than expected, I asked Martin to test
> > with a
> > > > > > simpler
> > > > > > > > > > process
> > > > > > > > > > > > > > >> > definition.
> > > > > > > > > > > > > > >> > > > One with just three nodes: start, script
> > and
> > > > > end.
> > > > > > > The
> > > > > > > > > > script
> > > > > > > > > > > > one
> > > > > > > > > > > > > > >> should
> > > > > > > > > > > > > > >> > > > change just one variable. This way we
> can
> > > > > analyze
> > > > > > if
> > > > > > > > the
> > > > > > > > > > > > number
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > >> > > queries
> > > > > > > > > > > > > > >> > > > is the expected one. From the single log
> > > > (audit
> > > > > > was
> > > > > > > > > > > activated
> > > > > > > > > > > > > > them)
> > > > > > > > > > > > > > >> my
> > > > > > > > > > > > > > >> > > > conclusion is that the number of
> > > > insert/updates
> > > > > > over
> > > > > > > > > > > processes
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > >> > nodes
> > > > > > > > > > > > > > >> > > > (there a lot over task, that I will
> > prefer to
> > > > > skip
> > > > > > > for
> > > > > > > > > > now,
> > > > > > > > > > > > baby
> > > > > > > > > > > > > > >> steps)
> > > > > > > > > > > > > > >> > > is
> > > > > > > > > > > > > > >> > > > the expected one.
> > > > > > > > > > > > > > >> > > > 2) Analysing the thread dump, we see
> > around 15
> > > > > > > threads
> > > > > > > > > > > > executing
> > > > > > > > > > > > > > >> this
> > > > > > > > > > > > > > >> > > line
> > > > > > > > > > > > > > >> > > > at
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > > > > > > > > > > > > >> > > > so its pretty clear the code to be
> > optimized
> > > > ;).
> > > > > > I'm
> > > > > > > > > > > > evaluating
> > > > > > > > > > > > > > >> > > > possibilities within JPA/Hibernate, but
> > I'm
> > > > > > starting
> > > > > > > > to
> > > > > > > > > > > think
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > >> it
> > > > > > > > > > > > > > >> > > might
> > > > > > > > > > > > > > >> > > > be better to switch to JDBC and skip
> > > > hibernate.
> > > > > > Our
> > > > > > > > > lives
> > > > > > > > > > > will
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > >> > > simpler,
> > > > > > > > > > > > > > >> > > > especially with a schema relatively
> simple
> > > > like
> > > > > > ours
> > > > > > > > > (that
> > > > > > > > > > > > will
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > >> my
> > > > > > > > > > > > > > >> > > > recommendation if I was an external
> > > > consultant)
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor
> > Zimányi <
> > > > > > > > > > > > > tzimanyi@apache.org
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > > > > Hi,
> > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > >> > > > > this will be a bit off-topic. However
> > as far
> > > > > as
> > > > > > > > > > > > performance, I
> > > > > > > > > > > > > > >> think
> > > > > > > > > > > > > > >> > we
> > > > > > > > > > > > > > >> > > > > should think about that we have string
> > > > primary
> > > > > > > keys
> > > > > > > > > > > (IDs). I
> > > > > > > > > > > > > > would
> > > > > > > > > > > > > > >> > > expect
> > > > > > > > > > > > > > >> > > > > the database systems are much better
> > with
> > > > > > indexing
> > > > > > > > > > numeric
> > > > > > > > > > > > > keys
> > > > > > > > > > > > > > >> than
> > > > > > > > > > > > > > >> > > > > strings. I remember from the past,
> when
> > I
> > > > was
> > > > > > > > working
> > > > > > > > > > with
> > > > > > > > > > > > > DBs,
> > > > > > > > > > > > > > >> that
> > > > > > > > > > > > > > >> > > > using
> > > > > > > > > > > > > > >> > > > > strings as keys or indexes was a
> > discouraged
> > > > > > > > practice.
> > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > >> > > > > Best regards,
> > > > > > > > > > > > > > >> > > > > Tibor
> > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > >> > > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler
> > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > napísal(a):
> > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > >> > > > > > I changed the test to use MongoDB
> [1]
> > and
> > > > I
> > > > > > > don't
> > > > > > > > > see
> > > > > > > > > > a
> > > > > > > > > > > > > > >> performance
> > > > > > > > > > > > > > >> > > > > > degradation with this setup [2].
> > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > >> > > > > > Please keep us posted of your
> > findings.
> > > > > > Thanks!
> > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > >> > > > > > Martin
> > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > >> > > > > > [1]
> > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > > >
> > https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > > > > > > > > > > > >> > > > > > [2]
> > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > >> > > > > >
> > ________________________________________
> > > > > > > > > > > > > > >> > > > > > From: Francisco Javier Tirado Sarti
> <
> > > > > > > > > > > ftirados@redhat.com>
> > > > > > > > > > > > > > >> > > > > > Sent: Wednesday, February 7, 2024
> > 11:40 AM
> > > > > > > > > > > > > > >> > > > > > To: dev@kie.apache.org
> > > > > > > > > > > > > > >> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION]
> > > > > > Performance
> > > > > > > > > > issues
> > > > > > > > > > > > with
> > > > > > > > > > > > > > >> > > data-index
> > > > > > > > > > > > > > >> > > > > > persistence addon
> > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > >> > > > > > yes, it can be index degradation
> > because
> > > > of
> > > > > > > size,
> > > > > > > > > but
> > > > > > > > > > I
> > > > > > > > > > > > > > believe
> > > > > > > > > > > > > > >> (I
> > > > > > > > > > > > > > >> > > > might
> > > > > > > > > > > > > > >> > > > > be
> > > > > > > > > > > > > > >> > > > > > wrong) the db is too small (yet) for
> > that.
> > > > > > > > > > > > > > >> > > > > > But, eventually, Postgres, when the
> > DB is
> > > > > huge
> > > > > > > > > enough,
> > > > > > > > > > > > > > >> unavoidably
> > > > > > > > > > > > > > >> > > will
> > > > > > > > > > > > > > >> > > > > > behave like the graphic that Martin
> > sent.
> > > > > > > > > > > > > > >> > > > > > Since I believe we are not huge
> enough
> > > > > (yet),
> > > > > > > lets
> > > > > > > > > > rule
> > > > > > > > > > > > out
> > > > > > > > > > > > > > >> another
> > > > > > > > > > > > > > >> > > > issue
> > > > > > > > > > > > > > >> > > > > > by analysing the sql logs (I
> requested
> > > > those
> > > > > > to
> > > > > > > > > Martin
> > > > > > > > > > > > > offline
> > > > > > > > > > > > > > >> and
> > > > > > > > > > > > > > >> > he
> > > > > > > > > > > > > > >> > > > is
> > > > > > > > > > > > > > >> > > > > > going to kindly collect them).
> > > > > > > > > > > > > > >> > > > > > Also Im curious to know if Mongo
> > behave in
> > > > > the
> > > > > > > > same
> > > > > > > > > > way.
> > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > >> > > > > > On Wed, Feb 7, 2024 at 7:25 PM
> Enrique
> > > > > > Gonzalez
> > > > > > > > > > > Martinez <
> > > > > > > > > > > > > > >> > > > > > egonzalez@apache.org> wrote:
> > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > >> > > > > > > Hi Francisco,
> > > > > > > > > > > > > > >> > > > > > > I would highly recommend to check
> > > > indexes
> > > > > > and
> > > > > > > > how
> > > > > > > > > > the
> > > > > > > > > > > > > > updates
> > > > > > > > > > > > > > >> > work
> > > > > > > > > > > > > > >> > > in
> > > > > > > > > > > > > > >> > > > > > data
> > > > > > > > > > > > > > >> > > > > > > index to avoid full scan table and
> > lock
> > > > > the
> > > > > > > full
> > > > > > > > > > > table.
> > > > > > > > > > > > > Some
> > > > > > > > > > > > > > >> db
> > > > > > > > > > > > > > >> > are
> > > > > > > > > > > > > > >> > > > > very
> > > > > > > > > > > > > > >> > > > > > > sensitive to that.
> > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > >> > > > > > > El mié, 7 feb 2024, 18:41,
> Francisco
> > > > > Javier
> > > > > > > > Tirado
> > > > > > > > > > > > Sarti <
> > > > > > > > > > > > > > >> > > > > > > ftirados@redhat.com> escribió:
> > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > Hi Martin,
> > > > > > > > > > > > > > >> > > > > > > > While I analyze the data, let me
> > ask
> > > > you
> > > > > > if
> > > > > > > it
> > > > > > > > > is
> > > > > > > > > > > > > possible
> > > > > > > > > > > > > > >> to
> > > > > > > > > > > > > > >> > > > perform
> > > > > > > > > > > > > > >> > > > > > > > another check (similar in a way
> to
> > > > > > disabling
> > > > > > > > > > > > data-index
> > > > > > > > > > > > > > like
> > > > > > > > > > > > > > >> > you
> > > > > > > > > > > > > > >> > > > do)
> > > > > > > > > > > > > > >> > > > > > Can
> > > > > > > > > > > > > > >> > > > > > > > you switch to MongoDB
> persistence
> > and
> > > > > > check
> > > > > > > if
> > > > > > > > > the
> > > > > > > > > > > > same
> > > > > > > > > > > > > > >> > > degradation
> > > > > > > > > > > > > > >> > > > > > that
> > > > > > > > > > > > > > >> > > > > > > is
> > > > > > > > > > > > > > >> > > > > > > > there for postgres remains?
> > > > > > > > > > > > > > >> > > > > > > > I do not know if this is
> feasible
> > but
> > > > > will
> > > > > > > > > > certainly
> > > > > > > > > > > > > > >> indicate
> > > > > > > > > > > > > > >> > the
> > > > > > > > > > > > > > >> > > > > > problem
> > > > > > > > > > > > > > >> > > > > > > > is on the postgres storage layer
> > and I
> > > > > do
> > > > > > > not
> > > > > > > > > > have a
> > > > > > > > > > > > > clear
> > > > > > > > > > > > > > >> > > > prediction
> > > > > > > > > > > > > > >> > > > > > of
> > > > > > > > > > > > > > >> > > > > > > > what we will see when doing this
> > > > switch.
> > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM
> > Martin
> > > > > > Weiler
> > > > > > > > > > > > > > >> > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > >> > > > > > > > wrote:
> > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > Hi Francisco,
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > thanks for your work on this
> > > > important
> > > > > > > > topic!
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > I would like to share some
> test
> > > > > results
> > > > > > > > here,
> > > > > > > > > > > which
> > > > > > > > > > > > > > might
> > > > > > > > > > > > > > >> > help
> > > > > > > > > > > > > > >> > > to
> > > > > > > > > > > > > > >> > > > > > > improve
> > > > > > > > > > > > > > >> > > > > > > > > the codebase even further. I
> am
> > > > using
> > > > > > the
> > > > > > > > > jmeter
> > > > > > > > > > > > based
> > > > > > > > > > > > > > >> test
> > > > > > > > > > > > > > >> > > case
> > > > > > > > > > > > > > >> > > > > from
> > > > > > > > > > > > > > >> > > > > > > > Pere
> > > > > > > > > > > > > > >> > > > > > > > > and Enrique (thanks guys!) [1]
> > which
> > > > > > uses
> > > > > > > a
> > > > > > > > > load
> > > > > > > > > > > of
> > > > > > > > > > > > 30
> > > > > > > > > > > > > > >> > threads
> > > > > > > > > > > > > > >> > > to
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > 1) start a new process
> instance
> > > > (POST)
> > > > > > > > > > > > > > >> > > > > > > > > 2) retrieve tasks for a user
> > (GET)
> > > > > > > > > > > > > > >> > > > > > > > > 3) fetches task details (GET)
> > > > > > > > > > > > > > >> > > > > > > > > 4) complete a task (POST)
> > > > > > > > > > > > > > >> > > > > > > > > 5) execute a query on
> data-audit
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > With this test setup, I
> noticed
> > that
> > > > > the
> > > > > > > > > > > performance
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > >> > > POST
> > > > > > > > > > > > > > >> > > > > > > > > requests, in particular the
> one
> > to
> > > > > > start a
> > > > > > > > new
> > > > > > > > > > > > process
> > > > > > > > > > > > > > >> > > instance,
> > > > > > > > > > > > > > >> > > > > > > degrades
> > > > > > > > > > > > > > >> > > > > > > > > over time - see graph [2]. If
> I
> > run
> > > > > the
> > > > > > > same
> > > > > > > > > > test
> > > > > > > > > > > > > > without
> > > > > > > > > > > > > > >> > > > > data-index,
> > > > > > > > > > > > > > >> > > > > > > > then
> > > > > > > > > > > > > > >> > > > > > > > > there is no such performance
> > > > > degradation
> > > > > > > > [3].
> > > > > > > > > > You
> > > > > > > > > > > > can
> > > > > > > > > > > > > > >> find a
> > > > > > > > > > > > > > >> > > > thread
> > > > > > > > > > > > > > >> > > > > > > dump
> > > > > > > > > > > > > > >> > > > > > > > > captured a few minutes into
> the
> > > > first
> > > > > > test
> > > > > > > > > here
> > > > > > > > > > > [4]
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > >> > might
> > > > > > > > > > > > > > >> > > > help
> > > > > > > > > > > > > > >> > > > > > to
> > > > > > > > > > > > > > >> > > > > > > > see
> > > > > > > > > > > > > > >> > > > > > > > > some of the contention points.
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > I'd appreciate if you could
> > take a
> > > > > look
> > > > > > > and
> > > > > > > > > see
> > > > > > > > > > if
> > > > > > > > > > > > > there
> > > > > > > > > > > > > > >> is
> > > > > > > > > > > > > > >> > > > > something
> > > > > > > > > > > > > > >> > > > > > > > that
> > > > > > > > > > > > > > >> > > > > > > > > can be further improved based
> on
> > > > your
> > > > > > > > previous
> > > > > > > > > > > work.
> > > > > > > > > > > > > If
> > > > > > > > > > > > > > >> you
> > > > > > > > > > > > > > >> > > need
> > > > > > > > > > > > > > >> > > > > any
> > > > > > > > > > > > > > >> > > > > > > > > additional data, let me know,
> > but
> > > > > > > otherwise
> > > > > > > > it
> > > > > > > > > > is
> > > > > > > > > > > > > > >> > > straightforward
> > > > > > > > > > > > > > >> > > > > to
> > > > > > > > > > > > > > >> > > > > > > run
> > > > > > > > > > > > > > >> > > > > > > > > the jmeter test as well.
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > Thanks,
> > > > > > > > > > > > > > >> > > > > > > > > Martin
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > [1]
> > > > > > > > > > > > > > >>
> > > > > https://github.com/pefernan/job-service-refactor-test/
> > > > > > > > > > > > > > >> > > > > > > > > [2]
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > > > > > > > > > > >> > > > > > > > > [3]
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > > > > > > > > > > >> > > > > > > > > [4]
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > ________________________________________
> > > > > > > > > > > > > > >> > > > > > > > > From: Francisco Javier Tirado
> > Sarti
> > > > <
> > > > > > > > > > > > > > ftirados@redhat.com>
> > > > > > > > > > > > > > >> > > > > > > > > Sent: Wednesday, January 17,
> > 2024
> > > > 9:13
> > > > > > AM
> > > > > > > > > > > > > > >> > > > > > > > > To: dev@kie.apache.org
> > > > > > > > > > > > > > >> > > > > > > > > Cc: Pere Fernandez Perez
> > > > > > > > > > > > > > >> > > > > > > > > Subject: [EXTERNAL] Re:
> > [DISCUSSION]
> > > > > > > > > Performance
> > > > > > > > > > > > > issues
> > > > > > > > > > > > > > >> with
> > > > > > > > > > > > > > >> > > > > > data-index
> > > > > > > > > > > > > > >> > > > > > > > > persistence addon
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > Hi Alex,
> > > > > > > > > > > > > > >> > > > > > > > > I did not take times (which
> > depends
> > > > > on a
> > > > > > > > > number
> > > > > > > > > > of
> > > > > > > > > > > > > > >> variables
> > > > > > > > > > > > > > >> > > that
> > > > > > > > > > > > > > >> > > > > > > > > drastically change between
> > > > > > environments),
> > > > > > > > but
> > > > > > > > > > > verify
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > >> > > > > number
> > > > > > > > > > > > > > >> > > > > > of
> > > > > > > > > > > > > > >> > > > > > > > > updates has been reduced
> > drastically
> > > > > > > without
> > > > > > > > > > > losing
> > > > > > > > > > > > > > >> > > > functionality,
> > > > > > > > > > > > > > >> > > > > > > which
> > > > > > > > > > > > > > >> > > > > > > > is
> > > > > > > > > > > > > > >> > > > > > > > > objectively a good thing. If
> > before
> > > > > the
> > > > > > > > > change,
> > > > > > > > > > > for
> > > > > > > > > > > > > > every
> > > > > > > > > > > > > > >> > node
> > > > > > > > > > > > > > >> > > > > > > executed,
> > > > > > > > > > > > > > >> > > > > > > > we
> > > > > > > > > > > > > > >> > > > > > > > > have an update for every node
> > > > > previously
> > > > > > > > > > executed,
> > > > > > > > > > > > so
> > > > > > > > > > > > > > if a
> > > > > > > > > > > > > > >> > > > process
> > > > > > > > > > > > > > >> > > > > > have
> > > > > > > > > > > > > > >> > > > > > > > 50
> > > > > > > > > > > > > > >> > > > > > > > > nodes to execute, we were
> > performing
> > > > > > > nearly
> > > > > > > > > > > 50*51/2
> > > > > > > > > > > > > > >> updates,
> > > > > > > > > > > > > > >> > > > which
> > > > > > > > > > > > > > >> > > > > > > gives
> > > > > > > > > > > > > > >> > > > > > > > us
> > > > > > > > > > > > > > >> > > > > > > > > a total of  1275 updates, now
> we
> > > > have
> > > > > > just
> > > > > > > > one
> > > > > > > > > > for
> > > > > > > > > > > > > every
> > > > > > > > > > > > > > >> node
> > > > > > > > > > > > > > >> > > > being
> > > > > > > > > > > > > > >> > > > > > > > > executed, implying a total of
> 50
> > > > > > updates.
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > On Wed, Jan 17, 2024 at
> 3:18 PM
> > Alex
> > > > > > > > Porcelli
> > > > > > > > > <
> > > > > > > > > > > > > > >> > > alex@porcelli.me>
> > > > > > > > > > > > > > >> > > > > > > wrote:
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > Francisco,
> > > > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > I noticed that your PR has
> > been
> > > > > > merged,
> > > > > > > > but
> > > > > > > > > I
> > > > > > > > > > > was
> > > > > > > > > > > > > > >> expecting
> > > > > > > > > > > > > > >> > > (at
> > > > > > > > > > > > > > >> > > > > > least
> > > > > > > > > > > > > > >> > > > > > > > > > was my understanding from
> this
> > > > > thread)
> > > > > > > > that
> > > > > > > > > > > before
> > > > > > > > > > > > > > >> merging
> > > > > > > > > > > > > > >> > > some
> > > > > > > > > > > > > > >> > > > > > > > > > benchmark data would be
> > shared in
> > > > > > > advance
> > > > > > > > -
> > > > > > > > > to
> > > > > > > > > > > > > assess
> > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > >> > > > > > > cost/benefit
> > > > > > > > > > > > > > >> > > > > > > > > > of such a decent size
> change.
> > > > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > Do you have any information
> to
> > > > > share?
> > > > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > On Sat, Dec 23, 2023 at
> > 4:02 AM
> > > > > > > Francisco
> > > > > > > > > > Javier
> > > > > > > > > > > > > > Tirado
> > > > > > > > > > > > > > >> > Sarti
> > > > > > > > > > > > > > >> > > > > > > > > > <ft...@redhat.com>
> wrote:
> > > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > > Yes, as intended, now we
> > have
> > > > one
> > > > > > > select
> > > > > > > > > and
> > > > > > > > > > > one
> > > > > > > > > > > > > > >> > > > insert/update
> > > > > > > > > > > > > > >> > > > > > per
> > > > > > > > > > > > > > >> > > > > > > > node
> > > > > > > > > > > > > > >> > > > > > > > > > > event.
> > > > > > > > > > > > > > >> > > > > > > > > > > I moved the PR as ready
> for
> > > > review
> > > > > > and
> > > > > > > > > give
> > > > > > > > > > > > @Pere
> > > > > > > > > > > > > > >> > Fernandez
> > > > > > > > > > > > > > >> > > > > Perez
> > > > > > > > > > > > > > >> > > > > > > > > > > <pe...@redhat.com>
> > > > permission
> > > > > to
> > > > > > > the
> > > > > > > > > > > branch
> > > > > > > > > > > > so
> > > > > > > > > > > > > > he
> > > > > > > > > > > > > > >> can
> > > > > > > > > > > > > > >> > > > edit
> > > > > > > > > > > > > > >> > > > > it
> > > > > > > > > > > > > > >> > > > > > > in
> > > > > > > > > > > > > > >> > > > > > > > > the
> > > > > > > > > > > > > > >> > > > > > > > > > > next two weeks (Ill be on
> > PTO)
> > > > if
> > > > > > > > > desired,
> > > > > > > > > > > > before
> > > > > > > > > > > > > > >> > merging.
> > > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > > On Thu, Dec 21, 2023 at
> > 5:58 PM
> > > > > Alex
> > > > > > > > > > Porcelli
> > > > > > > > > > > <
> > > > > > > > > > > > > > >> > > > > alex@porcelli.me>
> > > > > > > > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > > > Cool, thank you
> Francisco!
> > > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > > > Did you manage to get
> some
> > > > > > > preliminary
> > > > > > > > > > data
> > > > > > > > > > > > > about
> > > > > > > > > > > > > > >> > > > > improvements?
> > > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > > > On Thu, Dec 21, 2023 at
> > > > 11:52 AM
> > > > > > > > > Francisco
> > > > > > > > > > > > > Javier
> > > > > > > > > > > > > > >> > Tirado
> > > > > > > > > > > > > > >> > > > > Sarti
> > > > > > > > > > > > > > >> > > > > > > > > > > > <ft...@redhat.com>
> > wrote:
> > > > > > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > > > > Yes, after some delay
> > > > because
> > > > > of
> > > > > > > > > > quarkus 3
> > > > > > > > > > > > > > >> migration.
> > > > > > > > > > > > > > >> > > Im
> > > > > > > > > > > > > > >> > > > > > > refining
> > > > > > > > > > > > > > >> > > > > > > > > > this
> > > > > > > > > > > > > > >> > > > > > > > > > > > > draft PR
> > > > > > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >
> > > > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > > > > On Thu, Dec 21, 2023
> at
> > > > > 5:48 PM
> > > > > > > Alex
> > > > > > > > > > > > Porcelli
> > > > > > > > > > > > > <
> > > > > > > > > > > > > > >> > > > > > > alex@porcelli.me>
> > > > > > > > > > > > > > >> > > > > > > > > > wrote:
> > > > > > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Any update or new
> > findings
> > > > > on
> > > > > > > this
> > > > > > > > > > > topic?
> > > > > > > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > > > > > On Tue, Nov 28, 2023
> > at
> > > > > > 8:38 AM
> > > > > > > > > > > Francisco
> > > > > > > > > > > > > > Javier
> > > > > > > > > > > > > > >> > > Tirado
> > > > > > > > > > > > > > >> > > > > > Sarti
> > > > > > > > > > > > > > >> > > > > > > > > > > > > > <
> ftirados@redhat.com>
> > > > > wrote:
> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > After considering
> > > > > different
> > > > > > > > > options
> > > > > > > > > > to
> > > > > > > > > > > > > > improve
> > > > > > > > > > > > > > >> > > > > > performance,
> > > > > > > > > > > > > > >> > > > > > > > we
> > > > > > > > > > > > > > >> > > > > > > > > > feel
> > > > > > > > > > > > > > >> > > > > > > > > > > > that
> > > > > > > > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > is time to
> > "partially"
> > > > > move
> > > > > > > away
> > > > > > > > > > from
> > > > > > > > > > > > the
> > > > > > > > > > > > > > >> current
> > > > > > > > > > > > > > >> > > Map
> > > > > > > > > > >
>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Enrique Gonzalez Martinez <eg...@apache.org>]

+1 alex

El jue, 22 feb 2024, 0:14, Alex Porcelli <al...@porcelli.me> escribió:

> I think I need to clarify what I mean by REST...
>
> The REST endpoint I have in mind would execute the necessary cleaning
> based on an input parameter (ie. instanced closed for more than X days
> or something like that).
>
> The use of cron is that an external cron could call the cleaning REST
> endpoint from time to time (once a month, on every sunday night, etc).
>
>
> On Wed, Feb 21, 2024 at 10:50 AM Francisco Javier Tirado Sarti
> <ft...@redhat.com> wrote:
> >
> > Hi Alex,
> > I'm wondering how cron and rest can live together.
> > With cron the responsibility to do the clean up relies on a process
> > external to the microservice accessing the db to be purged, while where
> > using REST API the process performing the clean up is typically (but not
> > necessarily) the same microservice that accesses the DB. In that sense,
> > REST API is similar to using properties (in both approaches the
> responsible
> > to clean up the data is the microservice), but they are different because
> > REST is more sophisticated (but compatible, implementation wise we can
> use
> > a ConfigSource, that is updated by the REST and read, as a property, by
> the
> > cleanup routine). So, probably I'm missing something but I see the cron
> > approach not reconcilable with the REST approach.
> > Anyway just a detail, I agree the purge approach should be similar for
> all
> > microservices
> >
> > On Wed, Feb 21, 2024 at 4:37 PM Alex Porcelli <po...@apache.org>
> wrote:
> >
> > > +1 for provide consistent behavior across the board
> > >
> > > my 2c:
> > >
> > > +1 for REST interface (preferred for external general management
> > > perspective)
> > > -1 for properties to auto clean; this can be achieved with cron and
> rest
> > > call
> > >
> > > Now about GraphQL: the platform today is heavily invested in such
> > > interface, so it might make sense. BUT I’d consider it in a second
> moment.
> > >
> > >
> > > On Wed, Feb 21, 2024 at 6:52 AM Francisco Javier Tirado Sarti <
> > > ftirados@redhat.com> wrote:
> > >
> > > > Hi Enrique,
> > > > If we configure such policies through properties, it won't be enough
> to
> > > > define a naming convention and rely on the target platform
> configuration
> > > > capabilities (SpringBoot or Quarkus)?.
> > > >
> > > > On Wed, Feb 21, 2024 at 12:36 PM Enrique Gonzalez Martinez <
> > > > egonzalez@apache.org> wrote:
> > > >
> > > > > Hi Francisco
> > > > >
> > > > > The discussion we need to have before how to achieve certain
> features,
> > > is
> > > > > the overall of the user experience. If you want to clean it up
> that way
> > > > it
> > > > > is fine by me. I have nothing against setting policy related to
> some
> > > sort
> > > > > of clean up for a process completed some time ago.
> > > > >
> > > > > As we discussed previously the data index is a snapshot of the last
> > > state
> > > > > of the process instance included completed, but nothing was said
> once
> > > > > completed when we need to clean that up so any policy is welcome.
> How
> > > to
> > > > > achieve that policy is something completely different.
> > > > >
> > > > > The main problem is how every microservice is exposing that API to
> the
> > > > end
> > > > > user or consumer (other system) and which system is going to be the
> > > > façade
> > > > > for complex deployments and operations. That is the discussion I
> was
> > > > > mentioning before.
> > > > >
> > > > > What I want to avoid is to set different policies among components
> and
> > > we
> > > > > should strive to be as much as possible to offer certain
> capabilities
> > > in
> > > > > the same fashion, e.g: clean up mechanism.
> > > > > In the same way I want to avoid making the current system more
> complex
> > > > than
> > > > > they are. So far they are aligned offering one simple
> responsibility
> > > but
> > > > I
> > > > > would be against mixing for instance job service with data index.
> > > > >
> > > > > El mié, 21 feb 2024 a las 12:13, Francisco Javier Tirado Sarti (<
> > > > > ftirados@redhat.com>) escribió:
> > > > >
> > > > > > Hi Enrique,
> > > > > > In the case of data index I think the data to be purged is
> finished
> > > > > process
> > > > > > instances (I do not think we should remove process instance that
> has
> > > > been
> > > > > > alive for ages, even if it is very likely they are not going to
> be
> > > ever
> > > > > > completed)
> > > > > > Once you delete those process instances, you also delete the
> > > associated
> > > > > > user tasks and jobs.
> > > > > > Therefore the problem is relatively simple, to be able to
> configure
> > > how
> > > > > > much time a completed process instance should remain in the data
> > > index
> > > > > > database. We can take a simple approach: a property with a min
> > > duration
> > > > > > that cannot be changed once data index is started; a slightly
> complex
> > > > > one:
> > > > > > the same property but watching it to react for changes; or the
> full
> > > > > suite:
> > > > > > an admin API to be able to change the policy at any moment.
> > > > > > I think this discussion is worth having.
> > > > > >
> > > > > > On Wed, Feb 21, 2024 at 6:15 AM Enrique Gonzalez Martinez <
> > > > > > egonzalez@apache.org> wrote:
> > > > > >
> > > > > > > Hi Martin, the main problem regarding the purge is because it
> is
> > > > still
> > > > > > > unclear the policy for what tech to use and the future
> components.
> > > > > > >
> > > > > > > Recently we had a discussion about proposing graphql for this
> sort
> > > of
> > > > > > admin
> > > > > > > tasks. So far for subsystems we have been using rest endpoints
> > > (like
> > > > > > update
> > > > > > > timers or modify human task or change processes). There is one
> > > > > exception
> > > > > > > which is the gateway that is pure graphql and somehow uses
> graphql
> > > > for
> > > > > > > everything making complex operations under the hood.
> > > > > > >
> > > > > > > This has somehow frozen the purge for data audit for a bit and
> the
> > > > > > proposal
> > > > > > > was to use rest endpoints to do the clean up in the component
> and
> > > > offer
> > > > > > the
> > > > > > > graphql counterpart in the gateway promoting it to a first
> class
> > > > > citizen
> > > > > > > component instead of having it embedded in the data index.
> > > > > > >
> > > > > > > I would suggest to come up at least with a policy first
> regarding
> > > the
> > > > > > > convention every component should address this.
> > > > > > >
> > > > > > >
> > > > > > > El mar, 20 feb 2024, 23:39, Martin Weiler
> <mweiler@ibm.com.invalid
> > > >
> > > > > > > escribió:
> > > > > > >
> > > > > > > > IMO, it is good to have this discussion around data sanity
> now
> > > > > instead
> > > > > > of
> > > > > > > > putting it off until later when data has already accumulated
> in
> > > > > > > production
> > > > > > > > environments.
> > > > > > > >
> > > > > > > > Based on the input here, we are dealing with three types of
> data:
> > > > > > > > 1. Runtime data - active instances only, engine cleans up the
> > > data
> > > > > > > > automatically at process instance end
> > > > > > > > 2. Historic log data - data created by data-audit intended
> for
> > > long
> > > > > > term
> > > > > > > > storage
> > > > > > > > 3. Data-index data - somehow this data falls in between the
> two
> > > > > > > > aforementioned categories, with the idea of the data being
> > > > "recent",
> > > > > > but
> > > > > > > > not restricted to active instances only
> > > > > > > >
> > > > > > > > We'd need purge strategies for both #2 and #3 (perhaps
> different
> > > > > ones,
> > > > > > or
> > > > > > > > with different config settings) in order to prevent unlimited
> > > data
> > > > > > > growth.
> > > > > > > >
> > > > > > > > ________________________________________
> > > > > > > > From: Enrique Gonzalez Martinez <eg...@apache.org>
> > > > > > > > Sent: Monday, February 19, 2024 7:11 AM
> > > > > > > > To: dev@kie.apache.org
> > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > > > > data-index
> > > > > > > > persistence addon
> > > > > > > >
> > > > > > > > Hi Francisco,
> > > > > > > > To give you more context about this.
> > > > > > > >
> > > > > > > > STP is a concept, a process with certain constraints: no
> > > > persistence
> > > > > > and
> > > > > > > > returning the outcome in the call (sync execution with no
> idle
> > > > > states).
> > > > > > > It
> > > > > > > > was a requirement from a user in the past. One of the
> > > requirements
> > > > > was
> > > > > > > > leaving no trail. In v7 was easy because you could disable
> the
> > > > audit
> > > > > in
> > > > > > > > that case. Actually we have the same way to do what we did
> in v7
> > > in
> > > > > > here
> > > > > > > as
> > > > > > > > you can add/remove index just removing deps.
> > > > > > > >
> > > > > > > > We have the same outcome with different approaches and STP is
> > > > already
> > > > > > > > delivered.
> > > > > > > >
> > > > > > > > El lun, 19 feb 2024 a las 14:46, Francisco Javier Tirado
> Sarti (<
> > > > > > > > ftirados@redhat.com>) escribió:
> > > > > > > >
> > > > > > > > > Regarding STP (which is not a concept that we have in the
> > > code. I
> > > > > > mean
> > > > > > > > STP
> > > > > > > > > are processes as nonSTP are), I guess, as all processes,
> they
> > > > were
> > > > > > kept
> > > > > > > > in
> > > > > > > > > DataIndex once completed because users wanted (and still
> wants)
> > > > to
> > > > > > > check
> > > > > > > > > the result once the call had been performed. If we want to
> > > leave
> > > > no
> > > > > > > trace
> > > > > > > > > of them in DataIndex for some reason, we will need to make
> it a
> > > > > > > > > Runtimes concept so DataIndex can handle them in a
> different
> > > way.
> > > > > > > > >
> > > > > > > > > On Mon, Feb 19, 2024 at 2:27 PM Enrique Gonzalez Martinez <
> > > > > > > > > egonzalez@apache.org> wrote:
> > > > > > > > >
> > > > > > > > > > Alex:
> > > > > > > > > > Right now the data index is working in the same way as
> it did
> > > > in
> > > > > v7
> > > > > > > > with
> > > > > > > > > > the emitters. The only difference between two impl is
> that in
> > > > > here
> > > > > > > the
> > > > > > > > > > storage is pgsql instead elastic search.  You are right
> > > > regarding
> > > > > > is
> > > > > > > a
> > > > > > > > > > snapshot of the last state of the process but we did
> never
> > > > define
> > > > > > how
> > > > > > > > > long
> > > > > > > > > > would be alive that dats Honestly i am happy right now
> with
> > > the
> > > > > way
> > > > > > > it
> > > > > > > > > > works. The clean up mechanism is still tbd because we
> still
> > > > need
> > > > > to
> > > > > > > > > discuss
> > > > > > > > > > other stuff first.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Regarding stp is to leave no trail because u can get the
> > > > outcome
> > > > > > > > directly
> > > > > > > > > > from the call. It was defined like that in v7. So there
> is no
> > > > use
> > > > > > for
> > > > > > > > the
> > > > > > > > > > index or the audit.
> > > > > > > > > >
> > > > > > > > > > El lun, 19 feb 2024, 14:13, Francisco Javier Tirado
> Sarti <
> > > > > > > > > > ftirados@redhat.com> escribió:
> > > > > > > > > >
> > > > > > > > > > > Hi Alex,
> > > > > > > > > > > There has been some confusion about the purpose of
> > > DataIndex.
> > > > > To
> > > > > > be
> > > > > > > > > > honest
> > > > > > > > > > > I believe they were already sorted out, but your e-mail
> > > makes
> > > > > me
> > > > > > > > think
> > > > > > > > > > that
> > > > > > > > > > > is not the case ;). I let Kris to clarify that with
> you. My
> > > > > view
> > > > > > is
> > > > > > > > > that
> > > > > > > > > > > data-index is a way to query recently closed and active
> > > > > processes
> > > > > > > > (the
> > > > > > > > > > key
> > > > > > > > > > > here is the definition of recently, which in my opinion
> > > > should
> > > > > be
> > > > > > > > > > > configurable)
> > > > > > > > > > > But, besides that discussion and being pragmatic,
> keeping
> > > > > > finishing
> > > > > > > > > > process
> > > > > > > > > > > instances "for a while" in DataIndex was the only way
> for
> > > > users
> > > > > > to
> > > > > > > > > query
> > > > > > > > > > > the result of straight through processes. That's a
> function
> > > > > that
> > > > > > > > cannot
> > > > > > > > > > be
> > > > > > > > > > > removed right now
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Feb 19, 2024 at 1:33 PM Alex Porcelli <
> > > > > > porcelli@apache.org
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > if data index was supposed to provide snapshot view
> of
> > > the
> > > > > > > process
> > > > > > > > > > > > instance… why do we keep it after the process
> instance is
> > > > > > > finished?
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Feb 19, 2024 at 7:12 AM Francisco Javier
> Tirado
> > > > > Sarti <
> > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Martin.
> > > > > > > > > > > > > After taking a deeper look at this, I realize that
> the
> > > > > > > behaviour
> > > > > > > > is
> > > > > > > > > > the
> > > > > > > > > > > > > expected one.
> > > > > > > > > > > > > Runtimes DB does not track the completed process
> > > instance
> > > > > > > (that's
> > > > > > > > > > what
> > > > > > > > > > > > the
> > > > > > > > > > > > > JDBCProcessInstances warn is telling us), but
> > > DataIndex,
> > > > as
> > > > > > > > > expected,
> > > > > > > > > > > is
> > > > > > > > > > > > > tracking it in processes and nodes table. And yes
> it
> > > will
> > > > > > grow
> > > > > > > > over
> > > > > > > > > > > time.
> > > > > > > > > > > > > What we need is some configurable purge mechanism
> for
> > > > > > > DataIndex,
> > > > > > > > so
> > > > > > > > > > it
> > > > > > > > > > > > > eventually removes older completed process
> instances.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Feb 13, 2024 at 12:59 PM Francisco Javier
> > > Tirado
> > > > > > Sarti
> > > > > > > <
> > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Martin,
> > > > > > > > > > > > > > Good catch!. Looks like the skipping performed
> for
> > > > > process
> > > > > > > > > > instances
> > > > > > > > > > > is
> > > > > > > > > > > > > > not applied to node instances. Something we
> > > definitely
> > > > > need
> > > > > > > to
> > > > > > > > > > review
> > > > > > > > > > > > on
> > > > > > > > > > > > > > the runtimes side.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Feb 12, 2024 at 11:59 PM Martin Weiler
> > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> On a somewhat related note, testing a simple
> > > workflow
> > > > > > (start
> > > > > > > > ->
> > > > > > > > > > > script
> > > > > > > > > > > > > >> node -> end), I see the following messages in
> the
> > > > logs:
> > > > > > > > > > > > > >> 2024-02-12 22:49:50,493 28758dde544c WARN
> > > > > > > > > > > > > >>
> > > > > [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> > > > > > > > > > > > > >> (executor-thread-3) Skipping create of process
> > > > instance
> > > > > > id:
> > > > > > > > > > > > > >> 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> So far, so good. And I'd expect to see no trace
> of
> > > > this
> > > > > > > > process
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > >> database if I don't have data audit enabled.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> However, the 'processes' table contains a row
> with
> > > > > > state=2,
> > > > > > > > with
> > > > > > > > > > > > related
> > > > > > > > > > > > > >> entries in the 'nodes' table. In a load test, I
> see
> > > > > these
> > > > > > > > tables
> > > > > > > > > > > grow
> > > > > > > > > > > > > >> significantly over time. Am I missing something
> to
> > > > have
> > > > > > > these
> > > > > > > > > > > entries
> > > > > > > > > > > > > >> cleaned up automatically?
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> ________________________________________
> > > > > > > > > > > > > >> From: Martin Weiler <mw...@ibm.com.INVALID>
> > > > > > > > > > > > > >> Sent: Monday, February 12, 2024 3:40 PM
> > > > > > > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > > > > > > >> Subject: [EXTERNAL] RE: [DISCUSSION] Performance
> > > > issues
> > > > > > with
> > > > > > > > > > > > data-index
> > > > > > > > > > > > > >> persistence addon
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Thanks everyone for your input. Based on this
> > > > > discussion,
> > > > > > I
> > > > > > > > > opened
> > > > > > > > > > > the
> > > > > > > > > > > > > >> following PR:
> > > > > > > > > > > > > >>
> > > > > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1985
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> With this change, the performance seems to be
> stable
> > > > > over
> > > > > > > > time:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Martin
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> ________________________________________
> > > > > > > > > > > > > >> From: Gonzalo Muñoz <gm...@apache.org>
> > > > > > > > > > > > > >> Sent: Friday, February 9, 2024 9:42 AM
> > > > > > > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > > > > > > >> Subject: [EXTERNAL] Re: [DISCUSSION] Performance
> > > > issues
> > > > > > with
> > > > > > > > > > > > data-index
> > > > > > > > > > > > > >> persistence addon
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Great work Francisco,
> > > > > > > > > > > > > >> Martin, take a look at this link with some
> related
> > > > tips
> > > > > > (in
> > > > > > > > case
> > > > > > > > > > you
> > > > > > > > > > > > > find
> > > > > > > > > > > > > >> it useful):
> > > > > > > > > > > > > >>
> > > > > > > >
> https://www.cybertec-postgresql.com/en/index-your-foreign-key/
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> El vie, 9 feb 2024 a las 17:20, Francisco Javier
> > > > Tirado
> > > > > > > Sarti
> > > > > > > > (<
> > > > > > > > > > > > > >> ftirados@redhat.com>) escribió:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> > For the moment being, we will keep JPA till we
> > > > exhaust
> > > > > > all
> > > > > > > > > > > > > >> possibilities,
> > > > > > > > > > > > > >> > let's call switching from jpa to jdbc our
> hidden
> > > > plan
> > > > > B
> > > > > > ;)
> > > > > > > > > > > > > >> > I already told Martin, but in order everyone
> to
> > > > know,
> > > > > > just
> > > > > > > > > after
> > > > > > > > > > > > > writing
> > > > > > > > > > > > > >> > the previous email, I thought "what if
> Postgres is
> > > > not
> > > > > > > > > > > automatically
> > > > > > > > > > > > > >> > indexing foreign keys like mysql?" and, eureka
> > > > > > > > > > > > > >> > Postgres doc
> > > > > > > > > > > > > >> >
> > > > > > > > https://www.postgresql.org/docs/current/ddl-constraints.html
> > > > > > > > > > > > > >> > Mysql doc
> > > > > > > > > > > > > >> >
> > > > > > > > > > >
> > > > > > >
> > > https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> > > > > > > > > > > > > >> > These are the relevant excerpt
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > *Postgresql*
> > > > > > > > > > > > > >> > *A foreign key must reference columns that
> either
> > > > are
> > > > > a
> > > > > > > > > primary
> > > > > > > > > > > key
> > > > > > > > > > > > or
> > > > > > > > > > > > > >> form
> > > > > > > > > > > > > >> > a unique constraint, or are columns from a
> > > > non-partial
> > > > > > > > unique
> > > > > > > > > > > index.
> > > > > > > > > > > > > >> This
> > > > > > > > > > > > > >> > means that the referenced columns always have
> an
> > > > index
> > > > > > to
> > > > > > > > > allow
> > > > > > > > > > > > > >> efficient
> > > > > > > > > > > > > >> > lookups on whether a referencing row has a
> match.
> > > > > Since
> > > > > > a
> > > > > > > > > DELETE
> > > > > > > > > > > of
> > > > > > > > > > > > a
> > > > > > > > > > > > > >> row
> > > > > > > > > > > > > >> > from the referenced table or an UPDATE of a
> > > > referenced
> > > > > > > > column
> > > > > > > > > > will
> > > > > > > > > > > > > >> require
> > > > > > > > > > > > > >> > a scan of the referencing table for rows
> matching
> > > > the
> > > > > > old
> > > > > > > > > value,
> > > > > > > > > > > it
> > > > > > > > > > > > is
> > > > > > > > > > > > > >> > often a good idea to index the referencing
> columns
> > > > > too.
> > > > > > > > > Because
> > > > > > > > > > > this
> > > > > > > > > > > > > is
> > > > > > > > > > > > > >> not
> > > > > > > > > > > > > >> > always needed, and there are many choices
> > > available
> > > > on
> > > > > > how
> > > > > > > > to
> > > > > > > > > > > index,
> > > > > > > > > > > > > the
> > > > > > > > > > > > > >> > declaration of a foreign key constraint does
> not
> > > > > > > > automatically
> > > > > > > > > > > > create
> > > > > > > > > > > > > an
> > > > > > > > > > > > > >> > index on the referencing columns.*
> > > > > > > > > > > > > >> > *Mysql*
> > > > > > > > > > > > > >> > *MySQL requires that foreign key columns be
> > > indexed;
> > > > > if
> > > > > > > you
> > > > > > > > > > > create a
> > > > > > > > > > > > > >> table
> > > > > > > > > > > > > >> > with a foreign key constraint but no index on
> a
> > > > given
> > > > > > > > column,
> > > > > > > > > an
> > > > > > > > > > > > index
> > > > > > > > > > > > > >> is
> > > > > > > > > > > > > >> > created. *
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > So I asked Martin to especially create an
> index
> > > for
> > > > > > > > > > > > > process_instance_id
> > > > > > > > > > > > > >> > column on nodes table
> > > > > > > > > > > > > >> > I think that will fix the problem detected on
> the
> > > > > thread
> > > > > > > > dump.
> > > > > > > > > > > > > >> > The simpler process test to verify queries are
> > > fine
> > > > > > still
> > > > > > > > > > stands,
> > > > > > > > > > > > > >> though ;)
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <
> > > > > > > > > > tzimanyi@apache.org
> > > > > > > > > > > >
> > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > > I always preferred pure JDBC over Hibernate
> > > > myself,
> > > > > > just
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > > > sake
> > > > > > > > > > > > > >> of
> > > > > > > > > > > > > >> > > control of what is happening :) So I would
> not
> > > -1
> > > > > that
> > > > > > > > > myself.
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > Tibor
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > Dňa pi 9. 2. 2024, 17:00 Francisco Javier
> Tirado
> > > > > > Sarti <
> > > > > > > > > > > > > >> > > ftirados@redhat.com>
> > > > > > > > > > > > > >> > > napísal(a):
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > > Hi,
> > > > > > > > > > > > > >> > > > Usually I do not want to talk about work
> in
> > > > > progress
> > > > > > > > > because
> > > > > > > > > > > > > >> > preliminary
> > > > > > > > > > > > > >> > > > conclusions are pretty volatile but, well,
> > > there
> > > > > > are a
> > > > > > > > > > couple
> > > > > > > > > > > of
> > > > > > > > > > > > > >> things
> > > > > > > > > > > > > >> > > > that can be concluded from the really
> valuable
> > > > > > > > information
> > > > > > > > > > > that
> > > > > > > > > > > > > >> Martin
> > > > > > > > > > > > > >> > > > provided.
> > > > > > > > > > > > > >> > > > 1) In order to be able to determine if the
> > > > number
> > > > > of
> > > > > > > > > > > statements
> > > > > > > > > > > > is
> > > > > > > > > > > > > >> > larger
> > > > > > > > > > > > > >> > > > than expected, I asked Martin to test
> with a
> > > > > simpler
> > > > > > > > > process
> > > > > > > > > > > > > >> > definition.
> > > > > > > > > > > > > >> > > > One with just three nodes: start, script
> and
> > > > end.
> > > > > > The
> > > > > > > > > script
> > > > > > > > > > > one
> > > > > > > > > > > > > >> should
> > > > > > > > > > > > > >> > > > change just one variable. This way we can
> > > > analyze
> > > > > if
> > > > > > > the
> > > > > > > > > > > number
> > > > > > > > > > > > of
> > > > > > > > > > > > > >> > > queries
> > > > > > > > > > > > > >> > > > is the expected one. From the single log
> > > (audit
> > > > > was
> > > > > > > > > > activated
> > > > > > > > > > > > > them)
> > > > > > > > > > > > > >> my
> > > > > > > > > > > > > >> > > > conclusion is that the number of
> > > insert/updates
> > > > > over
> > > > > > > > > > processes
> > > > > > > > > > > > and
> > > > > > > > > > > > > >> > nodes
> > > > > > > > > > > > > >> > > > (there a lot over task, that I will
> prefer to
> > > > skip
> > > > > > for
> > > > > > > > > now,
> > > > > > > > > > > baby
> > > > > > > > > > > > > >> steps)
> > > > > > > > > > > > > >> > > is
> > > > > > > > > > > > > >> > > > the expected one.
> > > > > > > > > > > > > >> > > > 2) Analysing the thread dump, we see
> around 15
> > > > > > threads
> > > > > > > > > > > executing
> > > > > > > > > > > > > >> this
> > > > > > > > > > > > > >> > > line
> > > > > > > > > > > > > >> > > > at
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > > > > > > > > > > > >> > > > so its pretty clear the code to be
> optimized
> > > ;).
> > > > > I'm
> > > > > > > > > > > evaluating
> > > > > > > > > > > > > >> > > > possibilities within JPA/Hibernate, but
> I'm
> > > > > starting
> > > > > > > to
> > > > > > > > > > think
> > > > > > > > > > > > that
> > > > > > > > > > > > > >> it
> > > > > > > > > > > > > >> > > might
> > > > > > > > > > > > > >> > > > be better to switch to JDBC and skip
> > > hibernate.
> > > > > Our
> > > > > > > > lives
> > > > > > > > > > will
> > > > > > > > > > > > be
> > > > > > > > > > > > > >> > > simpler,
> > > > > > > > > > > > > >> > > > especially with a schema relatively simple
> > > like
> > > > > ours
> > > > > > > > (that
> > > > > > > > > > > will
> > > > > > > > > > > > be
> > > > > > > > > > > > > >> my
> > > > > > > > > > > > > >> > > > recommendation if I was an external
> > > consultant)
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor
> Zimányi <
> > > > > > > > > > > > tzimanyi@apache.org
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > > > > Hi,
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > >> > > > > this will be a bit off-topic. However
> as far
> > > > as
> > > > > > > > > > > performance, I
> > > > > > > > > > > > > >> think
> > > > > > > > > > > > > >> > we
> > > > > > > > > > > > > >> > > > > should think about that we have string
> > > primary
> > > > > > keys
> > > > > > > > > > (IDs). I
> > > > > > > > > > > > > would
> > > > > > > > > > > > > >> > > expect
> > > > > > > > > > > > > >> > > > > the database systems are much better
> with
> > > > > indexing
> > > > > > > > > numeric
> > > > > > > > > > > > keys
> > > > > > > > > > > > > >> than
> > > > > > > > > > > > > >> > > > > strings. I remember from the past, when
> I
> > > was
> > > > > > > working
> > > > > > > > > with
> > > > > > > > > > > > DBs,
> > > > > > > > > > > > > >> that
> > > > > > > > > > > > > >> > > > using
> > > > > > > > > > > > > >> > > > > strings as keys or indexes was a
> discouraged
> > > > > > > practice.
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > >> > > > > Best regards,
> > > > > > > > > > > > > >> > > > > Tibor
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > >> > > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler
> > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > napísal(a):
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > >> > > > > > I changed the test to use MongoDB [1]
> and
> > > I
> > > > > > don't
> > > > > > > > see
> > > > > > > > > a
> > > > > > > > > > > > > >> performance
> > > > > > > > > > > > > >> > > > > > degradation with this setup [2].
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > > > Please keep us posted of your
> findings.
> > > > > Thanks!
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > > > Martin
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > > > [1]
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > >
> https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > > > > > > > > > > >> > > > > > [2]
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > > >
> ________________________________________
> > > > > > > > > > > > > >> > > > > > From: Francisco Javier Tirado Sarti <
> > > > > > > > > > ftirados@redhat.com>
> > > > > > > > > > > > > >> > > > > > Sent: Wednesday, February 7, 2024
> 11:40 AM
> > > > > > > > > > > > > >> > > > > > To: dev@kie.apache.org
> > > > > > > > > > > > > >> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION]
> > > > > Performance
> > > > > > > > > issues
> > > > > > > > > > > with
> > > > > > > > > > > > > >> > > data-index
> > > > > > > > > > > > > >> > > > > > persistence addon
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > > > yes, it can be index degradation
> because
> > > of
> > > > > > size,
> > > > > > > > but
> > > > > > > > > I
> > > > > > > > > > > > > believe
> > > > > > > > > > > > > >> (I
> > > > > > > > > > > > > >> > > > might
> > > > > > > > > > > > > >> > > > > be
> > > > > > > > > > > > > >> > > > > > wrong) the db is too small (yet) for
> that.
> > > > > > > > > > > > > >> > > > > > But, eventually, Postgres, when the
> DB is
> > > > huge
> > > > > > > > enough,
> > > > > > > > > > > > > >> unavoidably
> > > > > > > > > > > > > >> > > will
> > > > > > > > > > > > > >> > > > > > behave like the graphic that Martin
> sent.
> > > > > > > > > > > > > >> > > > > > Since I believe we are not huge enough
> > > > (yet),
> > > > > > lets
> > > > > > > > > rule
> > > > > > > > > > > out
> > > > > > > > > > > > > >> another
> > > > > > > > > > > > > >> > > > issue
> > > > > > > > > > > > > >> > > > > > by analysing the sql logs (I requested
> > > those
> > > > > to
> > > > > > > > Martin
> > > > > > > > > > > > offline
> > > > > > > > > > > > > >> and
> > > > > > > > > > > > > >> > he
> > > > > > > > > > > > > >> > > > is
> > > > > > > > > > > > > >> > > > > > going to kindly collect them).
> > > > > > > > > > > > > >> > > > > > Also Im curious to know if Mongo
> behave in
> > > > the
> > > > > > > same
> > > > > > > > > way.
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique
> > > > > Gonzalez
> > > > > > > > > > Martinez <
> > > > > > > > > > > > > >> > > > > > egonzalez@apache.org> wrote:
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > > > > Hi Francisco,
> > > > > > > > > > > > > >> > > > > > > I would highly recommend to check
> > > indexes
> > > > > and
> > > > > > > how
> > > > > > > > > the
> > > > > > > > > > > > > updates
> > > > > > > > > > > > > >> > work
> > > > > > > > > > > > > >> > > in
> > > > > > > > > > > > > >> > > > > > data
> > > > > > > > > > > > > >> > > > > > > index to avoid full scan table and
> lock
> > > > the
> > > > > > full
> > > > > > > > > > table.
> > > > > > > > > > > > Some
> > > > > > > > > > > > > >> db
> > > > > > > > > > > > > >> > are
> > > > > > > > > > > > > >> > > > > very
> > > > > > > > > > > > > >> > > > > > > sensitive to that.
> > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > >> > > > > > > El mié, 7 feb 2024, 18:41, Francisco
> > > > Javier
> > > > > > > Tirado
> > > > > > > > > > > Sarti <
> > > > > > > > > > > > > >> > > > > > > ftirados@redhat.com> escribió:
> > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > >> > > > > > > > Hi Martin,
> > > > > > > > > > > > > >> > > > > > > > While I analyze the data, let me
> ask
> > > you
> > > > > if
> > > > > > it
> > > > > > > > is
> > > > > > > > > > > > possible
> > > > > > > > > > > > > >> to
> > > > > > > > > > > > > >> > > > perform
> > > > > > > > > > > > > >> > > > > > > > another check (similar in a way to
> > > > > disabling
> > > > > > > > > > > data-index
> > > > > > > > > > > > > like
> > > > > > > > > > > > > >> > you
> > > > > > > > > > > > > >> > > > do)
> > > > > > > > > > > > > >> > > > > > Can
> > > > > > > > > > > > > >> > > > > > > > you switch to MongoDB persistence
> and
> > > > > check
> > > > > > if
> > > > > > > > the
> > > > > > > > > > > same
> > > > > > > > > > > > > >> > > degradation
> > > > > > > > > > > > > >> > > > > > that
> > > > > > > > > > > > > >> > > > > > > is
> > > > > > > > > > > > > >> > > > > > > > there for postgres remains?
> > > > > > > > > > > > > >> > > > > > > > I do not know if this is feasible
> but
> > > > will
> > > > > > > > > certainly
> > > > > > > > > > > > > >> indicate
> > > > > > > > > > > > > >> > the
> > > > > > > > > > > > > >> > > > > > problem
> > > > > > > > > > > > > >> > > > > > > > is on the postgres storage layer
> and I
> > > > do
> > > > > > not
> > > > > > > > > have a
> > > > > > > > > > > > clear
> > > > > > > > > > > > > >> > > > prediction
> > > > > > > > > > > > > >> > > > > > of
> > > > > > > > > > > > > >> > > > > > > > what we will see when doing this
> > > switch.
> > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM
> Martin
> > > > > Weiler
> > > > > > > > > > > > > >> > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > > > > > wrote:
> > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > Hi Francisco,
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > thanks for your work on this
> > > important
> > > > > > > topic!
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > I would like to share some test
> > > > results
> > > > > > > here,
> > > > > > > > > > which
> > > > > > > > > > > > > might
> > > > > > > > > > > > > >> > help
> > > > > > > > > > > > > >> > > to
> > > > > > > > > > > > > >> > > > > > > improve
> > > > > > > > > > > > > >> > > > > > > > > the codebase even further. I am
> > > using
> > > > > the
> > > > > > > > jmeter
> > > > > > > > > > > based
> > > > > > > > > > > > > >> test
> > > > > > > > > > > > > >> > > case
> > > > > > > > > > > > > >> > > > > from
> > > > > > > > > > > > > >> > > > > > > > Pere
> > > > > > > > > > > > > >> > > > > > > > > and Enrique (thanks guys!) [1]
> which
> > > > > uses
> > > > > > a
> > > > > > > > load
> > > > > > > > > > of
> > > > > > > > > > > 30
> > > > > > > > > > > > > >> > threads
> > > > > > > > > > > > > >> > > to
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > 1) start a new process instance
> > > (POST)
> > > > > > > > > > > > > >> > > > > > > > > 2) retrieve tasks for a user
> (GET)
> > > > > > > > > > > > > >> > > > > > > > > 3) fetches task details (GET)
> > > > > > > > > > > > > >> > > > > > > > > 4) complete a task (POST)
> > > > > > > > > > > > > >> > > > > > > > > 5) execute a query on data-audit
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > With this test setup, I noticed
> that
> > > > the
> > > > > > > > > > performance
> > > > > > > > > > > > for
> > > > > > > > > > > > > >> the
> > > > > > > > > > > > > >> > > POST
> > > > > > > > > > > > > >> > > > > > > > > requests, in particular the one
> to
> > > > > start a
> > > > > > > new
> > > > > > > > > > > process
> > > > > > > > > > > > > >> > > instance,
> > > > > > > > > > > > > >> > > > > > > degrades
> > > > > > > > > > > > > >> > > > > > > > > over time - see graph [2]. If I
> run
> > > > the
> > > > > > same
> > > > > > > > > test
> > > > > > > > > > > > > without
> > > > > > > > > > > > > >> > > > > data-index,
> > > > > > > > > > > > > >> > > > > > > > then
> > > > > > > > > > > > > >> > > > > > > > > there is no such performance
> > > > degradation
> > > > > > > [3].
> > > > > > > > > You
> > > > > > > > > > > can
> > > > > > > > > > > > > >> find a
> > > > > > > > > > > > > >> > > > thread
> > > > > > > > > > > > > >> > > > > > > dump
> > > > > > > > > > > > > >> > > > > > > > > captured a few minutes into the
> > > first
> > > > > test
> > > > > > > > here
> > > > > > > > > > [4]
> > > > > > > > > > > > that
> > > > > > > > > > > > > >> > might
> > > > > > > > > > > > > >> > > > help
> > > > > > > > > > > > > >> > > > > > to
> > > > > > > > > > > > > >> > > > > > > > see
> > > > > > > > > > > > > >> > > > > > > > > some of the contention points.
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > I'd appreciate if you could
> take a
> > > > look
> > > > > > and
> > > > > > > > see
> > > > > > > > > if
> > > > > > > > > > > > there
> > > > > > > > > > > > > >> is
> > > > > > > > > > > > > >> > > > > something
> > > > > > > > > > > > > >> > > > > > > > that
> > > > > > > > > > > > > >> > > > > > > > > can be further improved based on
> > > your
> > > > > > > previous
> > > > > > > > > > work.
> > > > > > > > > > > > If
> > > > > > > > > > > > > >> you
> > > > > > > > > > > > > >> > > need
> > > > > > > > > > > > > >> > > > > any
> > > > > > > > > > > > > >> > > > > > > > > additional data, let me know,
> but
> > > > > > otherwise
> > > > > > > it
> > > > > > > > > is
> > > > > > > > > > > > > >> > > straightforward
> > > > > > > > > > > > > >> > > > > to
> > > > > > > > > > > > > >> > > > > > > run
> > > > > > > > > > > > > >> > > > > > > > > the jmeter test as well.
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > Thanks,
> > > > > > > > > > > > > >> > > > > > > > > Martin
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > [1]
> > > > > > > > > > > > > >>
> > > > https://github.com/pefernan/job-service-refactor-test/
> > > > > > > > > > > > > >> > > > > > > > > [2]
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > > > > > > > > > >> > > > > > > > > [3]
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > > > > > > > > > >> > > > > > > > > [4]
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > >
> > > > ________________________________________
> > > > > > > > > > > > > >> > > > > > > > > From: Francisco Javier Tirado
> Sarti
> > > <
> > > > > > > > > > > > > ftirados@redhat.com>
> > > > > > > > > > > > > >> > > > > > > > > Sent: Wednesday, January 17,
> 2024
> > > 9:13
> > > > > AM
> > > > > > > > > > > > > >> > > > > > > > > To: dev@kie.apache.org
> > > > > > > > > > > > > >> > > > > > > > > Cc: Pere Fernandez Perez
> > > > > > > > > > > > > >> > > > > > > > > Subject: [EXTERNAL] Re:
> [DISCUSSION]
> > > > > > > > Performance
> > > > > > > > > > > > issues
> > > > > > > > > > > > > >> with
> > > > > > > > > > > > > >> > > > > > data-index
> > > > > > > > > > > > > >> > > > > > > > > persistence addon
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > Hi Alex,
> > > > > > > > > > > > > >> > > > > > > > > I did not take times (which
> depends
> > > > on a
> > > > > > > > number
> > > > > > > > > of
> > > > > > > > > > > > > >> variables
> > > > > > > > > > > > > >> > > that
> > > > > > > > > > > > > >> > > > > > > > > drastically change between
> > > > > environments),
> > > > > > > but
> > > > > > > > > > verify
> > > > > > > > > > > > > that
> > > > > > > > > > > > > >> the
> > > > > > > > > > > > > >> > > > > number
> > > > > > > > > > > > > >> > > > > > of
> > > > > > > > > > > > > >> > > > > > > > > updates has been reduced
> drastically
> > > > > > without
> > > > > > > > > > losing
> > > > > > > > > > > > > >> > > > functionality,
> > > > > > > > > > > > > >> > > > > > > which
> > > > > > > > > > > > > >> > > > > > > > is
> > > > > > > > > > > > > >> > > > > > > > > objectively a good thing. If
> before
> > > > the
> > > > > > > > change,
> > > > > > > > > > for
> > > > > > > > > > > > > every
> > > > > > > > > > > > > >> > node
> > > > > > > > > > > > > >> > > > > > > executed,
> > > > > > > > > > > > > >> > > > > > > > we
> > > > > > > > > > > > > >> > > > > > > > > have an update for every node
> > > > previously
> > > > > > > > > executed,
> > > > > > > > > > > so
> > > > > > > > > > > > > if a
> > > > > > > > > > > > > >> > > > process
> > > > > > > > > > > > > >> > > > > > have
> > > > > > > > > > > > > >> > > > > > > > 50
> > > > > > > > > > > > > >> > > > > > > > > nodes to execute, we were
> performing
> > > > > > nearly
> > > > > > > > > > 50*51/2
> > > > > > > > > > > > > >> updates,
> > > > > > > > > > > > > >> > > > which
> > > > > > > > > > > > > >> > > > > > > gives
> > > > > > > > > > > > > >> > > > > > > > us
> > > > > > > > > > > > > >> > > > > > > > > a total of  1275 updates, now we
> > > have
> > > > > just
> > > > > > > one
> > > > > > > > > for
> > > > > > > > > > > > every
> > > > > > > > > > > > > >> node
> > > > > > > > > > > > > >> > > > being
> > > > > > > > > > > > > >> > > > > > > > > executed, implying a total of 50
> > > > > updates.
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM
> Alex
> > > > > > > Porcelli
> > > > > > > > <
> > > > > > > > > > > > > >> > > alex@porcelli.me>
> > > > > > > > > > > > > >> > > > > > > wrote:
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > Francisco,
> > > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > I noticed that your PR has
> been
> > > > > merged,
> > > > > > > but
> > > > > > > > I
> > > > > > > > > > was
> > > > > > > > > > > > > >> expecting
> > > > > > > > > > > > > >> > > (at
> > > > > > > > > > > > > >> > > > > > least
> > > > > > > > > > > > > >> > > > > > > > > > was my understanding from this
> > > > thread)
> > > > > > > that
> > > > > > > > > > before
> > > > > > > > > > > > > >> merging
> > > > > > > > > > > > > >> > > some
> > > > > > > > > > > > > >> > > > > > > > > > benchmark data would be
> shared in
> > > > > > advance
> > > > > > > -
> > > > > > > > to
> > > > > > > > > > > > assess
> > > > > > > > > > > > > >> the
> > > > > > > > > > > > > >> > > > > > > cost/benefit
> > > > > > > > > > > > > >> > > > > > > > > > of such a decent size change.
> > > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > Do you have any information to
> > > > share?
> > > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > On Sat, Dec 23, 2023 at
> 4:02 AM
> > > > > > Francisco
> > > > > > > > > Javier
> > > > > > > > > > > > > Tirado
> > > > > > > > > > > > > >> > Sarti
> > > > > > > > > > > > > >> > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > > Yes, as intended, now we
> have
> > > one
> > > > > > select
> > > > > > > > and
> > > > > > > > > > one
> > > > > > > > > > > > > >> > > > insert/update
> > > > > > > > > > > > > >> > > > > > per
> > > > > > > > > > > > > >> > > > > > > > node
> > > > > > > > > > > > > >> > > > > > > > > > > event.
> > > > > > > > > > > > > >> > > > > > > > > > > I moved the PR as ready for
> > > review
> > > > > and
> > > > > > > > give
> > > > > > > > > > > @Pere
> > > > > > > > > > > > > >> > Fernandez
> > > > > > > > > > > > > >> > > > > Perez
> > > > > > > > > > > > > >> > > > > > > > > > > <pe...@redhat.com>
> > > permission
> > > > to
> > > > > > the
> > > > > > > > > > branch
> > > > > > > > > > > so
> > > > > > > > > > > > > he
> > > > > > > > > > > > > >> can
> > > > > > > > > > > > > >> > > > edit
> > > > > > > > > > > > > >> > > > > it
> > > > > > > > > > > > > >> > > > > > > in
> > > > > > > > > > > > > >> > > > > > > > > the
> > > > > > > > > > > > > >> > > > > > > > > > > next two weeks (Ill be on
> PTO)
> > > if
> > > > > > > > desired,
> > > > > > > > > > > before
> > > > > > > > > > > > > >> > merging.
> > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > > On Thu, Dec 21, 2023 at
> 5:58 PM
> > > > Alex
> > > > > > > > > Porcelli
> > > > > > > > > > <
> > > > > > > > > > > > > >> > > > > alex@porcelli.me>
> > > > > > > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > > > Cool, thank you Francisco!
> > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > > > Did you manage to get some
> > > > > > preliminary
> > > > > > > > > data
> > > > > > > > > > > > about
> > > > > > > > > > > > > >> > > > > improvements?
> > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > > > On Thu, Dec 21, 2023 at
> > > 11:52 AM
> > > > > > > > Francisco
> > > > > > > > > > > > Javier
> > > > > > > > > > > > > >> > Tirado
> > > > > > > > > > > > > >> > > > > Sarti
> > > > > > > > > > > > > >> > > > > > > > > > > > <ft...@redhat.com>
> wrote:
> > > > > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > > > > Yes, after some delay
> > > because
> > > > of
> > > > > > > > > quarkus 3
> > > > > > > > > > > > > >> migration.
> > > > > > > > > > > > > >> > > Im
> > > > > > > > > > > > > >> > > > > > > refining
> > > > > > > > > > > > > >> > > > > > > > > > this
> > > > > > > > > > > > > >> > > > > > > > > > > > > draft PR
> > > > > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >
> > > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > > > > On Thu, Dec 21, 2023 at
> > > > 5:48 PM
> > > > > > Alex
> > > > > > > > > > > Porcelli
> > > > > > > > > > > > <
> > > > > > > > > > > > > >> > > > > > > alex@porcelli.me>
> > > > > > > > > > > > > >> > > > > > > > > > wrote:
> > > > > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > > > > > Any update or new
> findings
> > > > on
> > > > > > this
> > > > > > > > > > topic?
> > > > > > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > > > > > On Tue, Nov 28, 2023
> at
> > > > > 8:38 AM
> > > > > > > > > > Francisco
> > > > > > > > > > > > > Javier
> > > > > > > > > > > > > >> > > Tirado
> > > > > > > > > > > > > >> > > > > > Sarti
> > > > > > > > > > > > > >> > > > > > > > > > > > > > <ft...@redhat.com>
> > > > wrote:
> > > > > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > > > >> > > > > > > > > > > > > > > After considering
> > > > different
> > > > > > > > options
> > > > > > > > > to
> > > > > > > > > > > > > improve
> > > > > > > > > > > > > >> > > > > > performance,
> > > > > > > > > > > > > >> > > > > > > > we
> > > > > > > > > > > > > >> > > > > > > > > > feel
> > > > > > > > > > > > > >> > > > > > > > > > > > that
> > > > > > > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > > > > > > > >> > > > > > > > > > > > > > > is time to
> "partially"
> > > > move
> > > > > > away
> > > > > > > > > from
> > > > > > > > > > > the
> > > > > > > > > > > > > >> current
> > > > > > > > > > > > > >> > > Map
> > > > > > > > > >

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Alex Porcelli <al...@porcelli.me>]

I think I need to clarify what I mean by REST...

The REST endpoint I have in mind would execute the necessary cleaning
based on an input parameter (ie. instanced closed for more than X days
or something like that).

The use of cron is that an external cron could call the cleaning REST
endpoint from time to time (once a month, on every sunday night, etc).


On Wed, Feb 21, 2024 at 10:50 AM Francisco Javier Tirado Sarti
<ft...@redhat.com> wrote:
>
> Hi Alex,
> I'm wondering how cron and rest can live together.
> With cron the responsibility to do the clean up relies on a process
> external to the microservice accessing the db to be purged, while where
> using REST API the process performing the clean up is typically (but not
> necessarily) the same microservice that accesses the DB. In that sense,
> REST API is similar to using properties (in both approaches the responsible
> to clean up the data is the microservice), but they are different because
> REST is more sophisticated (but compatible, implementation wise we can use
> a ConfigSource, that is updated by the REST and read, as a property, by the
> cleanup routine). So, probably I'm missing something but I see the cron
> approach not reconcilable with the REST approach.
> Anyway just a detail, I agree the purge approach should be similar for all
> microservices
>
> On Wed, Feb 21, 2024 at 4:37 PM Alex Porcelli <po...@apache.org> wrote:
>
> > +1 for provide consistent behavior across the board
> >
> > my 2c:
> >
> > +1 for REST interface (preferred for external general management
> > perspective)
> > -1 for properties to auto clean; this can be achieved with cron and rest
> > call
> >
> > Now about GraphQL: the platform today is heavily invested in such
> > interface, so it might make sense. BUT I’d consider it in a second moment.
> >
> >
> > On Wed, Feb 21, 2024 at 6:52 AM Francisco Javier Tirado Sarti <
> > ftirados@redhat.com> wrote:
> >
> > > Hi Enrique,
> > > If we configure such policies through properties, it won't be enough to
> > > define a naming convention and rely on the target platform configuration
> > > capabilities (SpringBoot or Quarkus)?.
> > >
> > > On Wed, Feb 21, 2024 at 12:36 PM Enrique Gonzalez Martinez <
> > > egonzalez@apache.org> wrote:
> > >
> > > > Hi Francisco
> > > >
> > > > The discussion we need to have before how to achieve certain features,
> > is
> > > > the overall of the user experience. If you want to clean it up that way
> > > it
> > > > is fine by me. I have nothing against setting policy related to some
> > sort
> > > > of clean up for a process completed some time ago.
> > > >
> > > > As we discussed previously the data index is a snapshot of the last
> > state
> > > > of the process instance included completed, but nothing was said once
> > > > completed when we need to clean that up so any policy is welcome. How
> > to
> > > > achieve that policy is something completely different.
> > > >
> > > > The main problem is how every microservice is exposing that API to the
> > > end
> > > > user or consumer (other system) and which system is going to be the
> > > façade
> > > > for complex deployments and operations. That is the discussion I was
> > > > mentioning before.
> > > >
> > > > What I want to avoid is to set different policies among components and
> > we
> > > > should strive to be as much as possible to offer certain capabilities
> > in
> > > > the same fashion, e.g: clean up mechanism.
> > > > In the same way I want to avoid making the current system more complex
> > > than
> > > > they are. So far they are aligned offering one simple responsibility
> > but
> > > I
> > > > would be against mixing for instance job service with data index.
> > > >
> > > > El mié, 21 feb 2024 a las 12:13, Francisco Javier Tirado Sarti (<
> > > > ftirados@redhat.com>) escribió:
> > > >
> > > > > Hi Enrique,
> > > > > In the case of data index I think the data to be purged is finished
> > > > process
> > > > > instances (I do not think we should remove process instance that has
> > > been
> > > > > alive for ages, even if it is very likely they are not going to be
> > ever
> > > > > completed)
> > > > > Once you delete those process instances, you also delete the
> > associated
> > > > > user tasks and jobs.
> > > > > Therefore the problem is relatively simple, to be able to configure
> > how
> > > > > much time a completed process instance should remain in the data
> > index
> > > > > database. We can take a simple approach: a property with a min
> > duration
> > > > > that cannot be changed once data index is started; a slightly complex
> > > > one:
> > > > > the same property but watching it to react for changes; or the full
> > > > suite:
> > > > > an admin API to be able to change the policy at any moment.
> > > > > I think this discussion is worth having.
> > > > >
> > > > > On Wed, Feb 21, 2024 at 6:15 AM Enrique Gonzalez Martinez <
> > > > > egonzalez@apache.org> wrote:
> > > > >
> > > > > > Hi Martin, the main problem regarding the purge is because it is
> > > still
> > > > > > unclear the policy for what tech to use and the future components.
> > > > > >
> > > > > > Recently we had a discussion about proposing graphql for this sort
> > of
> > > > > admin
> > > > > > tasks. So far for subsystems we have been using rest endpoints
> > (like
> > > > > update
> > > > > > timers or modify human task or change processes). There is one
> > > > exception
> > > > > > which is the gateway that is pure graphql and somehow uses graphql
> > > for
> > > > > > everything making complex operations under the hood.
> > > > > >
> > > > > > This has somehow frozen the purge for data audit for a bit and the
> > > > > proposal
> > > > > > was to use rest endpoints to do the clean up in the component and
> > > offer
> > > > > the
> > > > > > graphql counterpart in the gateway promoting it to a first class
> > > > citizen
> > > > > > component instead of having it embedded in the data index.
> > > > > >
> > > > > > I would suggest to come up at least with a policy first regarding
> > the
> > > > > > convention every component should address this.
> > > > > >
> > > > > >
> > > > > > El mar, 20 feb 2024, 23:39, Martin Weiler <mweiler@ibm.com.invalid
> > >
> > > > > > escribió:
> > > > > >
> > > > > > > IMO, it is good to have this discussion around data sanity now
> > > > instead
> > > > > of
> > > > > > > putting it off until later when data has already accumulated in
> > > > > > production
> > > > > > > environments.
> > > > > > >
> > > > > > > Based on the input here, we are dealing with three types of data:
> > > > > > > 1. Runtime data - active instances only, engine cleans up the
> > data
> > > > > > > automatically at process instance end
> > > > > > > 2. Historic log data - data created by data-audit intended for
> > long
> > > > > term
> > > > > > > storage
> > > > > > > 3. Data-index data - somehow this data falls in between the two
> > > > > > > aforementioned categories, with the idea of the data being
> > > "recent",
> > > > > but
> > > > > > > not restricted to active instances only
> > > > > > >
> > > > > > > We'd need purge strategies for both #2 and #3 (perhaps different
> > > > ones,
> > > > > or
> > > > > > > with different config settings) in order to prevent unlimited
> > data
> > > > > > growth.
> > > > > > >
> > > > > > > ________________________________________
> > > > > > > From: Enrique Gonzalez Martinez <eg...@apache.org>
> > > > > > > Sent: Monday, February 19, 2024 7:11 AM
> > > > > > > To: dev@kie.apache.org
> > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > > > data-index
> > > > > > > persistence addon
> > > > > > >
> > > > > > > Hi Francisco,
> > > > > > > To give you more context about this.
> > > > > > >
> > > > > > > STP is a concept, a process with certain constraints: no
> > > persistence
> > > > > and
> > > > > > > returning the outcome in the call (sync execution with no idle
> > > > states).
> > > > > > It
> > > > > > > was a requirement from a user in the past. One of the
> > requirements
> > > > was
> > > > > > > leaving no trail. In v7 was easy because you could disable the
> > > audit
> > > > in
> > > > > > > that case. Actually we have the same way to do what we did in v7
> > in
> > > > > here
> > > > > > as
> > > > > > > you can add/remove index just removing deps.
> > > > > > >
> > > > > > > We have the same outcome with different approaches and STP is
> > > already
> > > > > > > delivered.
> > > > > > >
> > > > > > > El lun, 19 feb 2024 a las 14:46, Francisco Javier Tirado Sarti (<
> > > > > > > ftirados@redhat.com>) escribió:
> > > > > > >
> > > > > > > > Regarding STP (which is not a concept that we have in the
> > code. I
> > > > > mean
> > > > > > > STP
> > > > > > > > are processes as nonSTP are), I guess, as all processes, they
> > > were
> > > > > kept
> > > > > > > in
> > > > > > > > DataIndex once completed because users wanted (and still wants)
> > > to
> > > > > > check
> > > > > > > > the result once the call had been performed. If we want to
> > leave
> > > no
> > > > > > trace
> > > > > > > > of them in DataIndex for some reason, we will need to make it a
> > > > > > > > Runtimes concept so DataIndex can handle them in a different
> > way.
> > > > > > > >
> > > > > > > > On Mon, Feb 19, 2024 at 2:27 PM Enrique Gonzalez Martinez <
> > > > > > > > egonzalez@apache.org> wrote:
> > > > > > > >
> > > > > > > > > Alex:
> > > > > > > > > Right now the data index is working in the same way as it did
> > > in
> > > > v7
> > > > > > > with
> > > > > > > > > the emitters. The only difference between two impl is that in
> > > > here
> > > > > > the
> > > > > > > > > storage is pgsql instead elastic search.  You are right
> > > regarding
> > > > > is
> > > > > > a
> > > > > > > > > snapshot of the last state of the process but we did never
> > > define
> > > > > how
> > > > > > > > long
> > > > > > > > > would be alive that dats Honestly i am happy right now with
> > the
> > > > way
> > > > > > it
> > > > > > > > > works. The clean up mechanism is still tbd because we still
> > > need
> > > > to
> > > > > > > > discuss
> > > > > > > > > other stuff first.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Regarding stp is to leave no trail because u can get the
> > > outcome
> > > > > > > directly
> > > > > > > > > from the call. It was defined like that in v7. So there is no
> > > use
> > > > > for
> > > > > > > the
> > > > > > > > > index or the audit.
> > > > > > > > >
> > > > > > > > > El lun, 19 feb 2024, 14:13, Francisco Javier Tirado Sarti <
> > > > > > > > > ftirados@redhat.com> escribió:
> > > > > > > > >
> > > > > > > > > > Hi Alex,
> > > > > > > > > > There has been some confusion about the purpose of
> > DataIndex.
> > > > To
> > > > > be
> > > > > > > > > honest
> > > > > > > > > > I believe they were already sorted out, but your e-mail
> > makes
> > > > me
> > > > > > > think
> > > > > > > > > that
> > > > > > > > > > is not the case ;). I let Kris to clarify that with you. My
> > > > view
> > > > > is
> > > > > > > > that
> > > > > > > > > > data-index is a way to query recently closed and active
> > > > processes
> > > > > > > (the
> > > > > > > > > key
> > > > > > > > > > here is the definition of recently, which in my opinion
> > > should
> > > > be
> > > > > > > > > > configurable)
> > > > > > > > > > But, besides that discussion and being pragmatic, keeping
> > > > > finishing
> > > > > > > > > process
> > > > > > > > > > instances "for a while" in DataIndex was the only way for
> > > users
> > > > > to
> > > > > > > > query
> > > > > > > > > > the result of straight through processes. That's a function
> > > > that
> > > > > > > cannot
> > > > > > > > > be
> > > > > > > > > > removed right now
> > > > > > > > > >
> > > > > > > > > > On Mon, Feb 19, 2024 at 1:33 PM Alex Porcelli <
> > > > > porcelli@apache.org
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > if data index was supposed to provide snapshot view of
> > the
> > > > > > process
> > > > > > > > > > > instance… why do we keep it after the process instance is
> > > > > > finished?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Feb 19, 2024 at 7:12 AM Francisco Javier Tirado
> > > > Sarti <
> > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Martin.
> > > > > > > > > > > > After taking a deeper look at this, I realize that the
> > > > > > behaviour
> > > > > > > is
> > > > > > > > > the
> > > > > > > > > > > > expected one.
> > > > > > > > > > > > Runtimes DB does not track the completed process
> > instance
> > > > > > (that's
> > > > > > > > > what
> > > > > > > > > > > the
> > > > > > > > > > > > JDBCProcessInstances warn is telling us), but
> > DataIndex,
> > > as
> > > > > > > > expected,
> > > > > > > > > > is
> > > > > > > > > > > > tracking it in processes and nodes table. And yes it
> > will
> > > > > grow
> > > > > > > over
> > > > > > > > > > time.
> > > > > > > > > > > > What we need is some configurable purge mechanism for
> > > > > > DataIndex,
> > > > > > > so
> > > > > > > > > it
> > > > > > > > > > > > eventually removes older completed process instances.
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Feb 13, 2024 at 12:59 PM Francisco Javier
> > Tirado
> > > > > Sarti
> > > > > > <
> > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Martin,
> > > > > > > > > > > > > Good catch!. Looks like the skipping performed for
> > > > process
> > > > > > > > > instances
> > > > > > > > > > is
> > > > > > > > > > > > > not applied to node instances. Something we
> > definitely
> > > > need
> > > > > > to
> > > > > > > > > review
> > > > > > > > > > > on
> > > > > > > > > > > > > the runtimes side.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Feb 12, 2024 at 11:59 PM Martin Weiler
> > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> On a somewhat related note, testing a simple
> > workflow
> > > > > (start
> > > > > > > ->
> > > > > > > > > > script
> > > > > > > > > > > > >> node -> end), I see the following messages in the
> > > logs:
> > > > > > > > > > > > >> 2024-02-12 22:49:50,493 28758dde544c WARN
> > > > > > > > > > > > >>
> > > > [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> > > > > > > > > > > > >> (executor-thread-3) Skipping create of process
> > > instance
> > > > > id:
> > > > > > > > > > > > >> 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> So far, so good. And I'd expect to see no trace of
> > > this
> > > > > > > process
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > >> database if I don't have data audit enabled.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> However, the 'processes' table contains a row with
> > > > > state=2,
> > > > > > > with
> > > > > > > > > > > related
> > > > > > > > > > > > >> entries in the 'nodes' table. In a load test, I see
> > > > these
> > > > > > > tables
> > > > > > > > > > grow
> > > > > > > > > > > > >> significantly over time. Am I missing something to
> > > have
> > > > > > these
> > > > > > > > > > entries
> > > > > > > > > > > > >> cleaned up automatically?
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> ________________________________________
> > > > > > > > > > > > >> From: Martin Weiler <mw...@ibm.com.INVALID>
> > > > > > > > > > > > >> Sent: Monday, February 12, 2024 3:40 PM
> > > > > > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > > > > > >> Subject: [EXTERNAL] RE: [DISCUSSION] Performance
> > > issues
> > > > > with
> > > > > > > > > > > data-index
> > > > > > > > > > > > >> persistence addon
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Thanks everyone for your input. Based on this
> > > > discussion,
> > > > > I
> > > > > > > > opened
> > > > > > > > > > the
> > > > > > > > > > > > >> following PR:
> > > > > > > > > > > > >>
> > > > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1985
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> With this change, the performance seems to be stable
> > > > over
> > > > > > > time:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Martin
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> ________________________________________
> > > > > > > > > > > > >> From: Gonzalo Muñoz <gm...@apache.org>
> > > > > > > > > > > > >> Sent: Friday, February 9, 2024 9:42 AM
> > > > > > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > > > > > >> Subject: [EXTERNAL] Re: [DISCUSSION] Performance
> > > issues
> > > > > with
> > > > > > > > > > > data-index
> > > > > > > > > > > > >> persistence addon
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Great work Francisco,
> > > > > > > > > > > > >> Martin, take a look at this link with some related
> > > tips
> > > > > (in
> > > > > > > case
> > > > > > > > > you
> > > > > > > > > > > > find
> > > > > > > > > > > > >> it useful):
> > > > > > > > > > > > >>
> > > > > > > https://www.cybertec-postgresql.com/en/index-your-foreign-key/
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> El vie, 9 feb 2024 a las 17:20, Francisco Javier
> > > Tirado
> > > > > > Sarti
> > > > > > > (<
> > > > > > > > > > > > >> ftirados@redhat.com>) escribió:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> > For the moment being, we will keep JPA till we
> > > exhaust
> > > > > all
> > > > > > > > > > > > >> possibilities,
> > > > > > > > > > > > >> > let's call switching from jpa to jdbc our hidden
> > > plan
> > > > B
> > > > > ;)
> > > > > > > > > > > > >> > I already told Martin, but in order everyone to
> > > know,
> > > > > just
> > > > > > > > after
> > > > > > > > > > > > writing
> > > > > > > > > > > > >> > the previous email, I thought "what if Postgres is
> > > not
> > > > > > > > > > automatically
> > > > > > > > > > > > >> > indexing foreign keys like mysql?" and, eureka
> > > > > > > > > > > > >> > Postgres doc
> > > > > > > > > > > > >> >
> > > > > > > https://www.postgresql.org/docs/current/ddl-constraints.html
> > > > > > > > > > > > >> > Mysql doc
> > > > > > > > > > > > >> >
> > > > > > > > > >
> > > > > >
> > https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> > > > > > > > > > > > >> > These are the relevant excerpt
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > *Postgresql*
> > > > > > > > > > > > >> > *A foreign key must reference columns that either
> > > are
> > > > a
> > > > > > > > primary
> > > > > > > > > > key
> > > > > > > > > > > or
> > > > > > > > > > > > >> form
> > > > > > > > > > > > >> > a unique constraint, or are columns from a
> > > non-partial
> > > > > > > unique
> > > > > > > > > > index.
> > > > > > > > > > > > >> This
> > > > > > > > > > > > >> > means that the referenced columns always have an
> > > index
> > > > > to
> > > > > > > > allow
> > > > > > > > > > > > >> efficient
> > > > > > > > > > > > >> > lookups on whether a referencing row has a match.
> > > > Since
> > > > > a
> > > > > > > > DELETE
> > > > > > > > > > of
> > > > > > > > > > > a
> > > > > > > > > > > > >> row
> > > > > > > > > > > > >> > from the referenced table or an UPDATE of a
> > > referenced
> > > > > > > column
> > > > > > > > > will
> > > > > > > > > > > > >> require
> > > > > > > > > > > > >> > a scan of the referencing table for rows matching
> > > the
> > > > > old
> > > > > > > > value,
> > > > > > > > > > it
> > > > > > > > > > > is
> > > > > > > > > > > > >> > often a good idea to index the referencing columns
> > > > too.
> > > > > > > > Because
> > > > > > > > > > this
> > > > > > > > > > > > is
> > > > > > > > > > > > >> not
> > > > > > > > > > > > >> > always needed, and there are many choices
> > available
> > > on
> > > > > how
> > > > > > > to
> > > > > > > > > > index,
> > > > > > > > > > > > the
> > > > > > > > > > > > >> > declaration of a foreign key constraint does not
> > > > > > > automatically
> > > > > > > > > > > create
> > > > > > > > > > > > an
> > > > > > > > > > > > >> > index on the referencing columns.*
> > > > > > > > > > > > >> > *Mysql*
> > > > > > > > > > > > >> > *MySQL requires that foreign key columns be
> > indexed;
> > > > if
> > > > > > you
> > > > > > > > > > create a
> > > > > > > > > > > > >> table
> > > > > > > > > > > > >> > with a foreign key constraint but no index on a
> > > given
> > > > > > > column,
> > > > > > > > an
> > > > > > > > > > > index
> > > > > > > > > > > > >> is
> > > > > > > > > > > > >> > created. *
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > So I asked Martin to especially create an index
> > for
> > > > > > > > > > > > process_instance_id
> > > > > > > > > > > > >> > column on nodes table
> > > > > > > > > > > > >> > I think that will fix the problem detected on the
> > > > thread
> > > > > > > dump.
> > > > > > > > > > > > >> > The simpler process test to verify queries are
> > fine
> > > > > still
> > > > > > > > > stands,
> > > > > > > > > > > > >> though ;)
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <
> > > > > > > > > tzimanyi@apache.org
> > > > > > > > > > >
> > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > > I always preferred pure JDBC over Hibernate
> > > myself,
> > > > > just
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > > > sake
> > > > > > > > > > > > >> of
> > > > > > > > > > > > >> > > control of what is happening :) So I would not
> > -1
> > > > that
> > > > > > > > myself.
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > Tibor
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado
> > > > > Sarti <
> > > > > > > > > > > > >> > > ftirados@redhat.com>
> > > > > > > > > > > > >> > > napísal(a):
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > > Hi,
> > > > > > > > > > > > >> > > > Usually I do not want to talk about work in
> > > > progress
> > > > > > > > because
> > > > > > > > > > > > >> > preliminary
> > > > > > > > > > > > >> > > > conclusions are pretty volatile but, well,
> > there
> > > > > are a
> > > > > > > > > couple
> > > > > > > > > > of
> > > > > > > > > > > > >> things
> > > > > > > > > > > > >> > > > that can be concluded from the really valuable
> > > > > > > information
> > > > > > > > > > that
> > > > > > > > > > > > >> Martin
> > > > > > > > > > > > >> > > > provided.
> > > > > > > > > > > > >> > > > 1) In order to be able to determine if the
> > > number
> > > > of
> > > > > > > > > > statements
> > > > > > > > > > > is
> > > > > > > > > > > > >> > larger
> > > > > > > > > > > > >> > > > than expected, I asked Martin to test with a
> > > > simpler
> > > > > > > > process
> > > > > > > > > > > > >> > definition.
> > > > > > > > > > > > >> > > > One with just three nodes: start, script and
> > > end.
> > > > > The
> > > > > > > > script
> > > > > > > > > > one
> > > > > > > > > > > > >> should
> > > > > > > > > > > > >> > > > change just one variable. This way we can
> > > analyze
> > > > if
> > > > > > the
> > > > > > > > > > number
> > > > > > > > > > > of
> > > > > > > > > > > > >> > > queries
> > > > > > > > > > > > >> > > > is the expected one. From the single log
> > (audit
> > > > was
> > > > > > > > > activated
> > > > > > > > > > > > them)
> > > > > > > > > > > > >> my
> > > > > > > > > > > > >> > > > conclusion is that the number of
> > insert/updates
> > > > over
> > > > > > > > > processes
> > > > > > > > > > > and
> > > > > > > > > > > > >> > nodes
> > > > > > > > > > > > >> > > > (there a lot over task, that I will prefer to
> > > skip
> > > > > for
> > > > > > > > now,
> > > > > > > > > > baby
> > > > > > > > > > > > >> steps)
> > > > > > > > > > > > >> > > is
> > > > > > > > > > > > >> > > > the expected one.
> > > > > > > > > > > > >> > > > 2) Analysing the thread dump, we see around 15
> > > > > threads
> > > > > > > > > > executing
> > > > > > > > > > > > >> this
> > > > > > > > > > > > >> > > line
> > > > > > > > > > > > >> > > > at
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > > > > > > > > > > >> > > > so its pretty clear the code to be optimized
> > ;).
> > > > I'm
> > > > > > > > > > evaluating
> > > > > > > > > > > > >> > > > possibilities within JPA/Hibernate, but I'm
> > > > starting
> > > > > > to
> > > > > > > > > think
> > > > > > > > > > > that
> > > > > > > > > > > > >> it
> > > > > > > > > > > > >> > > might
> > > > > > > > > > > > >> > > > be better to switch to JDBC and skip
> > hibernate.
> > > > Our
> > > > > > > lives
> > > > > > > > > will
> > > > > > > > > > > be
> > > > > > > > > > > > >> > > simpler,
> > > > > > > > > > > > >> > > > especially with a schema relatively simple
> > like
> > > > ours
> > > > > > > (that
> > > > > > > > > > will
> > > > > > > > > > > be
> > > > > > > > > > > > >> my
> > > > > > > > > > > > >> > > > recommendation if I was an external
> > consultant)
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <
> > > > > > > > > > > tzimanyi@apache.org
> > > > > > > > > > > > >
> > > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > > Hi,
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > this will be a bit off-topic. However as far
> > > as
> > > > > > > > > > performance, I
> > > > > > > > > > > > >> think
> > > > > > > > > > > > >> > we
> > > > > > > > > > > > >> > > > > should think about that we have string
> > primary
> > > > > keys
> > > > > > > > > (IDs). I
> > > > > > > > > > > > would
> > > > > > > > > > > > >> > > expect
> > > > > > > > > > > > >> > > > > the database systems are much better with
> > > > indexing
> > > > > > > > numeric
> > > > > > > > > > > keys
> > > > > > > > > > > > >> than
> > > > > > > > > > > > >> > > > > strings. I remember from the past, when I
> > was
> > > > > > working
> > > > > > > > with
> > > > > > > > > > > DBs,
> > > > > > > > > > > > >> that
> > > > > > > > > > > > >> > > > using
> > > > > > > > > > > > >> > > > > strings as keys or indexes was a discouraged
> > > > > > practice.
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > Best regards,
> > > > > > > > > > > > >> > > > > Tibor
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler
> > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > napísal(a):
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > > I changed the test to use MongoDB [1] and
> > I
> > > > > don't
> > > > > > > see
> > > > > > > > a
> > > > > > > > > > > > >> performance
> > > > > > > > > > > > >> > > > > > degradation with this setup [2].
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > > > Please keep us posted of your findings.
> > > > Thanks!
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > > > Martin
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > > > [1]
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > > > > > > > > > >> > > > > > [2]
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > > > ________________________________________
> > > > > > > > > > > > >> > > > > > From: Francisco Javier Tirado Sarti <
> > > > > > > > > ftirados@redhat.com>
> > > > > > > > > > > > >> > > > > > Sent: Wednesday, February 7, 2024 11:40 AM
> > > > > > > > > > > > >> > > > > > To: dev@kie.apache.org
> > > > > > > > > > > > >> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION]
> > > > Performance
> > > > > > > > issues
> > > > > > > > > > with
> > > > > > > > > > > > >> > > data-index
> > > > > > > > > > > > >> > > > > > persistence addon
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > > > yes, it can be index degradation because
> > of
> > > > > size,
> > > > > > > but
> > > > > > > > I
> > > > > > > > > > > > believe
> > > > > > > > > > > > >> (I
> > > > > > > > > > > > >> > > > might
> > > > > > > > > > > > >> > > > > be
> > > > > > > > > > > > >> > > > > > wrong) the db is too small (yet) for that.
> > > > > > > > > > > > >> > > > > > But, eventually, Postgres, when the DB is
> > > huge
> > > > > > > enough,
> > > > > > > > > > > > >> unavoidably
> > > > > > > > > > > > >> > > will
> > > > > > > > > > > > >> > > > > > behave like the graphic that Martin sent.
> > > > > > > > > > > > >> > > > > > Since I believe we are not huge enough
> > > (yet),
> > > > > lets
> > > > > > > > rule
> > > > > > > > > > out
> > > > > > > > > > > > >> another
> > > > > > > > > > > > >> > > > issue
> > > > > > > > > > > > >> > > > > > by analysing the sql logs (I requested
> > those
> > > > to
> > > > > > > Martin
> > > > > > > > > > > offline
> > > > > > > > > > > > >> and
> > > > > > > > > > > > >> > he
> > > > > > > > > > > > >> > > > is
> > > > > > > > > > > > >> > > > > > going to kindly collect them).
> > > > > > > > > > > > >> > > > > > Also Im curious to know if Mongo behave in
> > > the
> > > > > > same
> > > > > > > > way.
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique
> > > > Gonzalez
> > > > > > > > > Martinez <
> > > > > > > > > > > > >> > > > > > egonzalez@apache.org> wrote:
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > > > > Hi Francisco,
> > > > > > > > > > > > >> > > > > > > I would highly recommend to check
> > indexes
> > > > and
> > > > > > how
> > > > > > > > the
> > > > > > > > > > > > updates
> > > > > > > > > > > > >> > work
> > > > > > > > > > > > >> > > in
> > > > > > > > > > > > >> > > > > > data
> > > > > > > > > > > > >> > > > > > > index to avoid full scan table and lock
> > > the
> > > > > full
> > > > > > > > > table.
> > > > > > > > > > > Some
> > > > > > > > > > > > >> db
> > > > > > > > > > > > >> > are
> > > > > > > > > > > > >> > > > > very
> > > > > > > > > > > > >> > > > > > > sensitive to that.
> > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > >> > > > > > > El mié, 7 feb 2024, 18:41, Francisco
> > > Javier
> > > > > > Tirado
> > > > > > > > > > Sarti <
> > > > > > > > > > > > >> > > > > > > ftirados@redhat.com> escribió:
> > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > >> > > > > > > > Hi Martin,
> > > > > > > > > > > > >> > > > > > > > While I analyze the data, let me ask
> > you
> > > > if
> > > > > it
> > > > > > > is
> > > > > > > > > > > possible
> > > > > > > > > > > > >> to
> > > > > > > > > > > > >> > > > perform
> > > > > > > > > > > > >> > > > > > > > another check (similar in a way to
> > > > disabling
> > > > > > > > > > data-index
> > > > > > > > > > > > like
> > > > > > > > > > > > >> > you
> > > > > > > > > > > > >> > > > do)
> > > > > > > > > > > > >> > > > > > Can
> > > > > > > > > > > > >> > > > > > > > you switch to MongoDB persistence and
> > > > check
> > > > > if
> > > > > > > the
> > > > > > > > > > same
> > > > > > > > > > > > >> > > degradation
> > > > > > > > > > > > >> > > > > > that
> > > > > > > > > > > > >> > > > > > > is
> > > > > > > > > > > > >> > > > > > > > there for postgres remains?
> > > > > > > > > > > > >> > > > > > > > I do not know if this is feasible but
> > > will
> > > > > > > > certainly
> > > > > > > > > > > > >> indicate
> > > > > > > > > > > > >> > the
> > > > > > > > > > > > >> > > > > > problem
> > > > > > > > > > > > >> > > > > > > > is on the postgres storage layer and I
> > > do
> > > > > not
> > > > > > > > have a
> > > > > > > > > > > clear
> > > > > > > > > > > > >> > > > prediction
> > > > > > > > > > > > >> > > > > > of
> > > > > > > > > > > > >> > > > > > > > what we will see when doing this
> > switch.
> > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > >> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin
> > > > Weiler
> > > > > > > > > > > > >> > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > > > > > wrote:
> > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > Hi Francisco,
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > thanks for your work on this
> > important
> > > > > > topic!
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > I would like to share some test
> > > results
> > > > > > here,
> > > > > > > > > which
> > > > > > > > > > > > might
> > > > > > > > > > > > >> > help
> > > > > > > > > > > > >> > > to
> > > > > > > > > > > > >> > > > > > > improve
> > > > > > > > > > > > >> > > > > > > > > the codebase even further. I am
> > using
> > > > the
> > > > > > > jmeter
> > > > > > > > > > based
> > > > > > > > > > > > >> test
> > > > > > > > > > > > >> > > case
> > > > > > > > > > > > >> > > > > from
> > > > > > > > > > > > >> > > > > > > > Pere
> > > > > > > > > > > > >> > > > > > > > > and Enrique (thanks guys!) [1] which
> > > > uses
> > > > > a
> > > > > > > load
> > > > > > > > > of
> > > > > > > > > > 30
> > > > > > > > > > > > >> > threads
> > > > > > > > > > > > >> > > to
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > 1) start a new process instance
> > (POST)
> > > > > > > > > > > > >> > > > > > > > > 2) retrieve tasks for a user (GET)
> > > > > > > > > > > > >> > > > > > > > > 3) fetches task details (GET)
> > > > > > > > > > > > >> > > > > > > > > 4) complete a task (POST)
> > > > > > > > > > > > >> > > > > > > > > 5) execute a query on data-audit
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > With this test setup, I noticed that
> > > the
> > > > > > > > > performance
> > > > > > > > > > > for
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >> > > POST
> > > > > > > > > > > > >> > > > > > > > > requests, in particular the one to
> > > > start a
> > > > > > new
> > > > > > > > > > process
> > > > > > > > > > > > >> > > instance,
> > > > > > > > > > > > >> > > > > > > degrades
> > > > > > > > > > > > >> > > > > > > > > over time - see graph [2]. If I run
> > > the
> > > > > same
> > > > > > > > test
> > > > > > > > > > > > without
> > > > > > > > > > > > >> > > > > data-index,
> > > > > > > > > > > > >> > > > > > > > then
> > > > > > > > > > > > >> > > > > > > > > there is no such performance
> > > degradation
> > > > > > [3].
> > > > > > > > You
> > > > > > > > > > can
> > > > > > > > > > > > >> find a
> > > > > > > > > > > > >> > > > thread
> > > > > > > > > > > > >> > > > > > > dump
> > > > > > > > > > > > >> > > > > > > > > captured a few minutes into the
> > first
> > > > test
> > > > > > > here
> > > > > > > > > [4]
> > > > > > > > > > > that
> > > > > > > > > > > > >> > might
> > > > > > > > > > > > >> > > > help
> > > > > > > > > > > > >> > > > > > to
> > > > > > > > > > > > >> > > > > > > > see
> > > > > > > > > > > > >> > > > > > > > > some of the contention points.
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > I'd appreciate if you could take a
> > > look
> > > > > and
> > > > > > > see
> > > > > > > > if
> > > > > > > > > > > there
> > > > > > > > > > > > >> is
> > > > > > > > > > > > >> > > > > something
> > > > > > > > > > > > >> > > > > > > > that
> > > > > > > > > > > > >> > > > > > > > > can be further improved based on
> > your
> > > > > > previous
> > > > > > > > > work.
> > > > > > > > > > > If
> > > > > > > > > > > > >> you
> > > > > > > > > > > > >> > > need
> > > > > > > > > > > > >> > > > > any
> > > > > > > > > > > > >> > > > > > > > > additional data, let me know, but
> > > > > otherwise
> > > > > > it
> > > > > > > > is
> > > > > > > > > > > > >> > > straightforward
> > > > > > > > > > > > >> > > > > to
> > > > > > > > > > > > >> > > > > > > run
> > > > > > > > > > > > >> > > > > > > > > the jmeter test as well.
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > Thanks,
> > > > > > > > > > > > >> > > > > > > > > Martin
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > [1]
> > > > > > > > > > > > >>
> > > https://github.com/pefernan/job-service-refactor-test/
> > > > > > > > > > > > >> > > > > > > > > [2]
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > > > > > > > > >> > > > > > > > > [3]
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > > > > > > > > >> > > > > > > > > [4]
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > >
> > > ________________________________________
> > > > > > > > > > > > >> > > > > > > > > From: Francisco Javier Tirado Sarti
> > <
> > > > > > > > > > > > ftirados@redhat.com>
> > > > > > > > > > > > >> > > > > > > > > Sent: Wednesday, January 17, 2024
> > 9:13
> > > > AM
> > > > > > > > > > > > >> > > > > > > > > To: dev@kie.apache.org
> > > > > > > > > > > > >> > > > > > > > > Cc: Pere Fernandez Perez
> > > > > > > > > > > > >> > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION]
> > > > > > > Performance
> > > > > > > > > > > issues
> > > > > > > > > > > > >> with
> > > > > > > > > > > > >> > > > > > data-index
> > > > > > > > > > > > >> > > > > > > > > persistence addon
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > Hi Alex,
> > > > > > > > > > > > >> > > > > > > > > I did not take times (which depends
> > > on a
> > > > > > > number
> > > > > > > > of
> > > > > > > > > > > > >> variables
> > > > > > > > > > > > >> > > that
> > > > > > > > > > > > >> > > > > > > > > drastically change between
> > > > environments),
> > > > > > but
> > > > > > > > > verify
> > > > > > > > > > > > that
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >> > > > > number
> > > > > > > > > > > > >> > > > > > of
> > > > > > > > > > > > >> > > > > > > > > updates has been reduced drastically
> > > > > without
> > > > > > > > > losing
> > > > > > > > > > > > >> > > > functionality,
> > > > > > > > > > > > >> > > > > > > which
> > > > > > > > > > > > >> > > > > > > > is
> > > > > > > > > > > > >> > > > > > > > > objectively a good thing. If before
> > > the
> > > > > > > change,
> > > > > > > > > for
> > > > > > > > > > > > every
> > > > > > > > > > > > >> > node
> > > > > > > > > > > > >> > > > > > > executed,
> > > > > > > > > > > > >> > > > > > > > we
> > > > > > > > > > > > >> > > > > > > > > have an update for every node
> > > previously
> > > > > > > > executed,
> > > > > > > > > > so
> > > > > > > > > > > > if a
> > > > > > > > > > > > >> > > > process
> > > > > > > > > > > > >> > > > > > have
> > > > > > > > > > > > >> > > > > > > > 50
> > > > > > > > > > > > >> > > > > > > > > nodes to execute, we were performing
> > > > > nearly
> > > > > > > > > 50*51/2
> > > > > > > > > > > > >> updates,
> > > > > > > > > > > > >> > > > which
> > > > > > > > > > > > >> > > > > > > gives
> > > > > > > > > > > > >> > > > > > > > us
> > > > > > > > > > > > >> > > > > > > > > a total of  1275 updates, now we
> > have
> > > > just
> > > > > > one
> > > > > > > > for
> > > > > > > > > > > every
> > > > > > > > > > > > >> node
> > > > > > > > > > > > >> > > > being
> > > > > > > > > > > > >> > > > > > > > > executed, implying a total of 50
> > > > updates.
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex
> > > > > > Porcelli
> > > > > > > <
> > > > > > > > > > > > >> > > alex@porcelli.me>
> > > > > > > > > > > > >> > > > > > > wrote:
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > Francisco,
> > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > I noticed that your PR has been
> > > > merged,
> > > > > > but
> > > > > > > I
> > > > > > > > > was
> > > > > > > > > > > > >> expecting
> > > > > > > > > > > > >> > > (at
> > > > > > > > > > > > >> > > > > > least
> > > > > > > > > > > > >> > > > > > > > > > was my understanding from this
> > > thread)
> > > > > > that
> > > > > > > > > before
> > > > > > > > > > > > >> merging
> > > > > > > > > > > > >> > > some
> > > > > > > > > > > > >> > > > > > > > > > benchmark data would be shared in
> > > > > advance
> > > > > > -
> > > > > > > to
> > > > > > > > > > > assess
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >> > > > > > > cost/benefit
> > > > > > > > > > > > >> > > > > > > > > > of such a decent size change.
> > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > Do you have any information to
> > > share?
> > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM
> > > > > Francisco
> > > > > > > > Javier
> > > > > > > > > > > > Tirado
> > > > > > > > > > > > >> > Sarti
> > > > > > > > > > > > >> > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > Yes, as intended, now we have
> > one
> > > > > select
> > > > > > > and
> > > > > > > > > one
> > > > > > > > > > > > >> > > > insert/update
> > > > > > > > > > > > >> > > > > > per
> > > > > > > > > > > > >> > > > > > > > node
> > > > > > > > > > > > >> > > > > > > > > > > event.
> > > > > > > > > > > > >> > > > > > > > > > > I moved the PR as ready for
> > review
> > > > and
> > > > > > > give
> > > > > > > > > > @Pere
> > > > > > > > > > > > >> > Fernandez
> > > > > > > > > > > > >> > > > > Perez
> > > > > > > > > > > > >> > > > > > > > > > > <pe...@redhat.com>
> > permission
> > > to
> > > > > the
> > > > > > > > > branch
> > > > > > > > > > so
> > > > > > > > > > > > he
> > > > > > > > > > > > >> can
> > > > > > > > > > > > >> > > > edit
> > > > > > > > > > > > >> > > > > it
> > > > > > > > > > > > >> > > > > > > in
> > > > > > > > > > > > >> > > > > > > > > the
> > > > > > > > > > > > >> > > > > > > > > > > next two weeks (Ill be on PTO)
> > if
> > > > > > > desired,
> > > > > > > > > > before
> > > > > > > > > > > > >> > merging.
> > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM
> > > Alex
> > > > > > > > Porcelli
> > > > > > > > > <
> > > > > > > > > > > > >> > > > > alex@porcelli.me>
> > > > > > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > Cool, thank you Francisco!
> > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > Did you manage to get some
> > > > > preliminary
> > > > > > > > data
> > > > > > > > > > > about
> > > > > > > > > > > > >> > > > > improvements?
> > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > On Thu, Dec 21, 2023 at
> > 11:52 AM
> > > > > > > Francisco
> > > > > > > > > > > Javier
> > > > > > > > > > > > >> > Tirado
> > > > > > > > > > > > >> > > > > Sarti
> > > > > > > > > > > > >> > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > > Yes, after some delay
> > because
> > > of
> > > > > > > > quarkus 3
> > > > > > > > > > > > >> migration.
> > > > > > > > > > > > >> > > Im
> > > > > > > > > > > > >> > > > > > > refining
> > > > > > > > > > > > >> > > > > > > > > > this
> > > > > > > > > > > > >> > > > > > > > > > > > > draft PR
> > > > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > >
> > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > > On Thu, Dec 21, 2023 at
> > > 5:48 PM
> > > > > Alex
> > > > > > > > > > Porcelli
> > > > > > > > > > > <
> > > > > > > > > > > > >> > > > > > > alex@porcelli.me>
> > > > > > > > > > > > >> > > > > > > > > > wrote:
> > > > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > > > Any update or new findings
> > > on
> > > > > this
> > > > > > > > > topic?
> > > > > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > > > On Tue, Nov 28, 2023 at
> > > > 8:38 AM
> > > > > > > > > Francisco
> > > > > > > > > > > > Javier
> > > > > > > > > > > > >> > > Tirado
> > > > > > > > > > > > >> > > > > > Sarti
> > > > > > > > > > > > >> > > > > > > > > > > > > > <ft...@redhat.com>
> > > wrote:
> > > > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > > >> > > > > > > > > > > > > > > After considering
> > > different
> > > > > > > options
> > > > > > > > to
> > > > > > > > > > > > improve
> > > > > > > > > > > > >> > > > > > performance,
> > > > > > > > > > > > >> > > > > > > > we
> > > > > > > > > > > > >> > > > > > > > > > feel
> > > > > > > > > > > > >> > > > > > > > > > > > that
> > > > > > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > > > > > > >> > > > > > > > > > > > > > > is time to "partially"
> > > move
> > > > > away
> > > > > > > > from
> > > > > > > > > > the
> > > > > > > > > > > > >> current
> > > > > > > > > > > > >> > > Map
> > > > > > > > > > > > >> > > > > > style
> > > > > > > > > > > > >> > > > > > > > > > > > interface (
> > > > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > > > > > > > >> > > > > > > > > > > > > > )
> > > > > > > > > > > > >> > > > > > > > > > > > > > > which was shared with
> > > > Trusty,
> > > > > to
> > > > > > > one
> > > > > > > > > > more
> > > > > > > > > > > > >> > suitable
> > > > > > > > > > > > >> > > > for
> > > > > > > > > > > > >> > > > > > > usage
> > > > > > > > > > > > >> > > > > > > > > > with a
> > > > > > > > > > > > >> > > > > > > > > > > > > > > relational DB like
> > > > postgresql
> > > > > > (but
> > > > > > > > > still
> > > > > > > > > > > > >> > compatible
> > > > > > > > > > > > >> > > > > with
> > > > > > > > > > > > >> > > > > > > big
> > > > > > > > > > > > >> > > > > > > > > > table
> > > > > > > > > > > > >> > > > > > > > > > > > dbs).
> > > > > > > > > > > > >> > > > > > > > > > > > > > > The idea will be to
> > > replace
> > > > > > > generic
> > > > > > > > > > > Storage
> > > > > > > > > > > > >> > > interface
> > > > > > > > > > > > >> > > > > by
> > > > > > > > > > > > >> > > > > > > four
> > > > > > > > > > > > >> > > > > > > > > > > > specific
> > > > > > > > > > > > >> > > > > > > > > > > > > > > interfaces (which will
> > > > inherit
> > > > > > > from
> > > > > > > > a
> > > > > > > > > > > common
> > > > > > > > > > > > >> one
> > > > > > > > > > > > >> > > that
> > > > > > > > > > > > >> > > > > > keeps
> > > > > > > > > > > > >> > > > > > > > the
> > > > > > > > > > > > >> > > > > > > > > > query
> > > > > > > > > > > > >> > > > > > > > > > > > > > part
> > > > > > > > > > > > >> > > > > > > > > > > > > > > at is it. with get and
> > > query
> > > > > > > > methods),
> > > > > > > > > > > that
> > > > > > > > > > > > >> will
> > > > > > > > > > > > >> > > > > include
> > > > > > > > > > > > >> > > > > > > the
> > > > > > > > > > > > >> > > > > > > > > > required
> > > > > > > > > > > > >> > > > > > > > > > > > > > > modification operations
> > > for
> > > > > the
> > > > > > > four
> > > > > > > > > > > > DataIndex
> > > > > > > > > > > > >> > > > > "domains":
> > > > > > > > > > > > >> > > > > > > > > > > > > > processinstance,
> > > > > > > > > > > > >> > > > > > > > > > > > > > > usertask,
> > > processdefinitions
> > > > > and
> > > > > > > > jobs.
> > > > > > > > > > > Those
> > > > > > > > > > > > >> > > > interfaces
> > > > > > > > > > > > >> > > > > > > will
> > > > > > > > > > > > >> > > > > > > > > > define
> > > > > > > > > > > > >> > > > > > > > > > > > > > methods
> > > > > > > > > > > > >> > > > > > > > > > > > > > > like addNode,
> > addVariable,
> > > > > > > > updateTask,
> > > > > > > > > > > > >> > > > > addAttachment.....
> > > > > > > > > > > > >> > > > > > > > that
> > > > > > > > > > > > >> > > > > > > > > > will
> > > > > > > > > > > > >> > > > > > > > > > > > allow
> > > > > > > > > > > > >> > > > > > > > > > > > > > > the persistent layer
> > > > > > > implementation
> > > > > > > > > to
> > > > > > > > > > > just
> > > > > > > > > > > > >> > update
> > > > > > > > > > > > >> > > > the
> > > > > > > > > > > > >> > > > > > > > needed
> > > > > > > > > > > > >> > > > > > > > > > info
> > > > > > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > > > > >> > > > > > > > > > > > > > > DB  (for example, for
> > > > addNode
> > > > > in
> > > > > > > > > > Postgres,
> > > > > > > > > > > > >> just
> > > > > > > > > > > > >> > > > insert
> > > > > > > > > > > > >> > > > > a
> > > > > > > > > > > > >> > > > > > > row
> > > > > > > > > > > > >> > > > > > > > > into
> > > > > > > > > > > > >> > > > > > > > > > > > nodes
> > > > > > > > > > > > >> > > > > > > > > > > > > > > table, for addNode in
> > > Mongo,
> > > > > > > > basically
> > > > > > > > > > the
> > > > > > > > > > > > >> same
> > > > > > > > > > > > >> > > > atomic
> > > > > > > > > > > > >> > > > > > > upsert
> > > > > > > > > > > > >> > > > > > > > > > > > operation
> > > > > > > > > > > > >> > > > > > > > > > > > > > > that is currently done).
> > > > > > > Therefore,
> > > > > > > > we
> > > > > > > > > > > > >> increase
> > > > > > > > > > > > >> > > > > > performance
> > > > > > > > > > > > >> > > > > > > > for
> > > > > > > > > > > > >> > > > > > > > > > > > Postgres
> > > > > > > > > > > > >> > > > > > > > > > > > > > > and keep the current one
> > > for
> > > > > > > Mongo.
> > > > > > > > > The
> > > > > > > > > > > > >> current
> > > > > > > > > > > > >> > DB
> > > > > > > > > > > > >> > > > > > schemas
> > > > > > > > > > > > >> > > > > > > > > won't
> > > > > > > > > > > > >> > > > > > > > > > be
> > > > > > > > > > > > >> > > > > > > > > > > > > > > touched.
> > > > > > > > > > > > >> > > > > > > > > > > > > > > Since the code change is
> > > > > large,
> > > > > > I
> > > > > > > do
> > > > > > > > > not
> > > > > > > > > > > > think
> > > > > > > > > > > > >> > I'll
> > > > > > > > > > > > >> > > > be
> > > > > > > > > > > > >> > > > > > able
> > > > > > > > > > > > >> > > > > > > > to
> > > > > > > > > > > > >> > > > > > > > > > have
> > > > > > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > > > > >> > > > > > > > > > > > > > PR
> > > > > > > > > > > > >> > > > > > > > > > > > > > > ready till next week.
> > > > > > > > > > > > >> > > > > > > > > > > > > > > But before starting,
> > > please
> > > > > let
> > > > > > me
> > > > > > > > > know
> > > > > > > > > > if
> > > > > > > > > > > > >> that
> > > > > > > > > > > > >> > > > > approach
> > > > > > > > > > > > >> > > > > > is
> > > > > > > > > > > > >> > > > > > > > > fine
> > > > > > > > > > > > >> > > > > > > > > > for
> > > > > > > > > > > > >> > > > > > > > > > > > you.
> > > > > > > > > > > > >> > > > > > > > > > > > > > > Best regards.
> > > > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> > > > > 6:55 PM
> > > > > > > Alex
> > > > > > > > > > > > Porcelli
> > > > > > > > > > > > >> <
> > > > > > > > > > > > >> > > > > > > > > alex@porcelli.me>
> > > > > > > > > > > > >> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > > > > > Thank you Francisco to
> > > > > getting
> > > > > > > > > deeper
> > > > > > > > > > on
> > > > > > > > > > > > >> this…
> > > > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > > > > > Looking forward to see
> > > the
> > > > > > > results
> > > > > > > > > of
> > > > > > > > > > > your
> > > > > > > > > > > > >> > > > suggested
> > > > > > > > > > > > >> > > > > > > > > > improvements.
> > > > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023
> > at
> > > > > > 9:40 AM
> > > > > > > > > > > Francisco
> > > > > > > > > > > > >> > Javier
> > > > > > > > > > > > >> > > > > Tirado
> > > > > > > > > > > > >> > > > > > > > > Sarti <
> > > > > > > > > > > > >> > > > > > > > > > > > > > > > ftirados@redhat.com>
> > > > wrote:
> > > > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > I forgot to attach
> > the
> > > > > > queries
> > > > > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > On Fri, Nov 24, 2023
> > > at
> > > > > > > 3:04 PM
> > > > > > > > > > > > Francisco
> > > > > > > > > > > > >> > > Javier
> > > > > > > > > > > > >> > > > > > Tirado
> > > > > > > > > > > > >> > > > > > > > > > Sarti <
> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > ftirados@redhat.com
> > >
> > > > > wrote:
> > > > > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> Hi,
> > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> A brief update on
> > > this
> > > > > > topic.
> > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> After doing a
> > simple
> > > > test
> > > > > > > with
> > > > > > > > > > > example
> > > > > > > > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > > > > > > >> > > > > > > > > > > > > > > > ,
> > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> the number of
> > updates
> > > > > over
> > > > > > > > Nodes
> > > > > > > > > > > table
> > > > > > > > > > > > is
> > > > > > > > > > > > >> > n*n,
> > > > > > > > > > > > >> > > > so
> > > > > > > > > > > > >> > > > > we
> > > > > > > > > > > > >> > > > > > > > > manage
> > > > > > > > > > > > >> > > > > > > > > > to
> > > > > > > > > > > > >> > > > > > > > > > > > > > obtain a
> > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> perfect quadratic
> > > > > > performance
> > > > > > > > > > > > >> degradation.
> > > > > > > > > > > > >> > The
> > > > > > > > > > > > >> > > > > > problem
> > > > > > > > > > > > >> > > > > > > > is
> > > > > > > > > > > > >> > > > > > > > > > worse
> > > > > > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > > > > >> > > > > > > > > > > > > > > > case
> > > > > >
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
For additional commands, e-mail: dev-help@kie.apache.org

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Francisco Javier Tirado Sarti <ft...@redhat.com>]

Hi Alex,
I'm wondering how cron and rest can live together.
With cron the responsibility to do the clean up relies on a process
external to the microservice accessing the db to be purged, while where
using REST API the process performing the clean up is typically (but not
necessarily) the same microservice that accesses the DB. In that sense,
REST API is similar to using properties (in both approaches the responsible
to clean up the data is the microservice), but they are different because
REST is more sophisticated (but compatible, implementation wise we can use
a ConfigSource, that is updated by the REST and read, as a property, by the
cleanup routine). So, probably I'm missing something but I see the cron
approach not reconcilable with the REST approach.
Anyway just a detail, I agree the purge approach should be similar for all
microservices

On Wed, Feb 21, 2024 at 4:37 PM Alex Porcelli <po...@apache.org> wrote:

> +1 for provide consistent behavior across the board
>
> my 2c:
>
> +1 for REST interface (preferred for external general management
> perspective)
> -1 for properties to auto clean; this can be achieved with cron and rest
> call
>
> Now about GraphQL: the platform today is heavily invested in such
> interface, so it might make sense. BUT I’d consider it in a second moment.
>
>
> On Wed, Feb 21, 2024 at 6:52 AM Francisco Javier Tirado Sarti <
> ftirados@redhat.com> wrote:
>
> > Hi Enrique,
> > If we configure such policies through properties, it won't be enough to
> > define a naming convention and rely on the target platform configuration
> > capabilities (SpringBoot or Quarkus)?.
> >
> > On Wed, Feb 21, 2024 at 12:36 PM Enrique Gonzalez Martinez <
> > egonzalez@apache.org> wrote:
> >
> > > Hi Francisco
> > >
> > > The discussion we need to have before how to achieve certain features,
> is
> > > the overall of the user experience. If you want to clean it up that way
> > it
> > > is fine by me. I have nothing against setting policy related to some
> sort
> > > of clean up for a process completed some time ago.
> > >
> > > As we discussed previously the data index is a snapshot of the last
> state
> > > of the process instance included completed, but nothing was said once
> > > completed when we need to clean that up so any policy is welcome. How
> to
> > > achieve that policy is something completely different.
> > >
> > > The main problem is how every microservice is exposing that API to the
> > end
> > > user or consumer (other system) and which system is going to be the
> > façade
> > > for complex deployments and operations. That is the discussion I was
> > > mentioning before.
> > >
> > > What I want to avoid is to set different policies among components and
> we
> > > should strive to be as much as possible to offer certain capabilities
> in
> > > the same fashion, e.g: clean up mechanism.
> > > In the same way I want to avoid making the current system more complex
> > than
> > > they are. So far they are aligned offering one simple responsibility
> but
> > I
> > > would be against mixing for instance job service with data index.
> > >
> > > El mié, 21 feb 2024 a las 12:13, Francisco Javier Tirado Sarti (<
> > > ftirados@redhat.com>) escribió:
> > >
> > > > Hi Enrique,
> > > > In the case of data index I think the data to be purged is finished
> > > process
> > > > instances (I do not think we should remove process instance that has
> > been
> > > > alive for ages, even if it is very likely they are not going to be
> ever
> > > > completed)
> > > > Once you delete those process instances, you also delete the
> associated
> > > > user tasks and jobs.
> > > > Therefore the problem is relatively simple, to be able to configure
> how
> > > > much time a completed process instance should remain in the data
> index
> > > > database. We can take a simple approach: a property with a min
> duration
> > > > that cannot be changed once data index is started; a slightly complex
> > > one:
> > > > the same property but watching it to react for changes; or the full
> > > suite:
> > > > an admin API to be able to change the policy at any moment.
> > > > I think this discussion is worth having.
> > > >
> > > > On Wed, Feb 21, 2024 at 6:15 AM Enrique Gonzalez Martinez <
> > > > egonzalez@apache.org> wrote:
> > > >
> > > > > Hi Martin, the main problem regarding the purge is because it is
> > still
> > > > > unclear the policy for what tech to use and the future components.
> > > > >
> > > > > Recently we had a discussion about proposing graphql for this sort
> of
> > > > admin
> > > > > tasks. So far for subsystems we have been using rest endpoints
> (like
> > > > update
> > > > > timers or modify human task or change processes). There is one
> > > exception
> > > > > which is the gateway that is pure graphql and somehow uses graphql
> > for
> > > > > everything making complex operations under the hood.
> > > > >
> > > > > This has somehow frozen the purge for data audit for a bit and the
> > > > proposal
> > > > > was to use rest endpoints to do the clean up in the component and
> > offer
> > > > the
> > > > > graphql counterpart in the gateway promoting it to a first class
> > > citizen
> > > > > component instead of having it embedded in the data index.
> > > > >
> > > > > I would suggest to come up at least with a policy first regarding
> the
> > > > > convention every component should address this.
> > > > >
> > > > >
> > > > > El mar, 20 feb 2024, 23:39, Martin Weiler <mweiler@ibm.com.invalid
> >
> > > > > escribió:
> > > > >
> > > > > > IMO, it is good to have this discussion around data sanity now
> > > instead
> > > > of
> > > > > > putting it off until later when data has already accumulated in
> > > > > production
> > > > > > environments.
> > > > > >
> > > > > > Based on the input here, we are dealing with three types of data:
> > > > > > 1. Runtime data - active instances only, engine cleans up the
> data
> > > > > > automatically at process instance end
> > > > > > 2. Historic log data - data created by data-audit intended for
> long
> > > > term
> > > > > > storage
> > > > > > 3. Data-index data - somehow this data falls in between the two
> > > > > > aforementioned categories, with the idea of the data being
> > "recent",
> > > > but
> > > > > > not restricted to active instances only
> > > > > >
> > > > > > We'd need purge strategies for both #2 and #3 (perhaps different
> > > ones,
> > > > or
> > > > > > with different config settings) in order to prevent unlimited
> data
> > > > > growth.
> > > > > >
> > > > > > ________________________________________
> > > > > > From: Enrique Gonzalez Martinez <eg...@apache.org>
> > > > > > Sent: Monday, February 19, 2024 7:11 AM
> > > > > > To: dev@kie.apache.org
> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > > data-index
> > > > > > persistence addon
> > > > > >
> > > > > > Hi Francisco,
> > > > > > To give you more context about this.
> > > > > >
> > > > > > STP is a concept, a process with certain constraints: no
> > persistence
> > > > and
> > > > > > returning the outcome in the call (sync execution with no idle
> > > states).
> > > > > It
> > > > > > was a requirement from a user in the past. One of the
> requirements
> > > was
> > > > > > leaving no trail. In v7 was easy because you could disable the
> > audit
> > > in
> > > > > > that case. Actually we have the same way to do what we did in v7
> in
> > > > here
> > > > > as
> > > > > > you can add/remove index just removing deps.
> > > > > >
> > > > > > We have the same outcome with different approaches and STP is
> > already
> > > > > > delivered.
> > > > > >
> > > > > > El lun, 19 feb 2024 a las 14:46, Francisco Javier Tirado Sarti (<
> > > > > > ftirados@redhat.com>) escribió:
> > > > > >
> > > > > > > Regarding STP (which is not a concept that we have in the
> code. I
> > > > mean
> > > > > > STP
> > > > > > > are processes as nonSTP are), I guess, as all processes, they
> > were
> > > > kept
> > > > > > in
> > > > > > > DataIndex once completed because users wanted (and still wants)
> > to
> > > > > check
> > > > > > > the result once the call had been performed. If we want to
> leave
> > no
> > > > > trace
> > > > > > > of them in DataIndex for some reason, we will need to make it a
> > > > > > > Runtimes concept so DataIndex can handle them in a different
> way.
> > > > > > >
> > > > > > > On Mon, Feb 19, 2024 at 2:27 PM Enrique Gonzalez Martinez <
> > > > > > > egonzalez@apache.org> wrote:
> > > > > > >
> > > > > > > > Alex:
> > > > > > > > Right now the data index is working in the same way as it did
> > in
> > > v7
> > > > > > with
> > > > > > > > the emitters. The only difference between two impl is that in
> > > here
> > > > > the
> > > > > > > > storage is pgsql instead elastic search.  You are right
> > regarding
> > > > is
> > > > > a
> > > > > > > > snapshot of the last state of the process but we did never
> > define
> > > > how
> > > > > > > long
> > > > > > > > would be alive that dats Honestly i am happy right now with
> the
> > > way
> > > > > it
> > > > > > > > works. The clean up mechanism is still tbd because we still
> > need
> > > to
> > > > > > > discuss
> > > > > > > > other stuff first.
> > > > > > > >
> > > > > > > >
> > > > > > > > Regarding stp is to leave no trail because u can get the
> > outcome
> > > > > > directly
> > > > > > > > from the call. It was defined like that in v7. So there is no
> > use
> > > > for
> > > > > > the
> > > > > > > > index or the audit.
> > > > > > > >
> > > > > > > > El lun, 19 feb 2024, 14:13, Francisco Javier Tirado Sarti <
> > > > > > > > ftirados@redhat.com> escribió:
> > > > > > > >
> > > > > > > > > Hi Alex,
> > > > > > > > > There has been some confusion about the purpose of
> DataIndex.
> > > To
> > > > be
> > > > > > > > honest
> > > > > > > > > I believe they were already sorted out, but your e-mail
> makes
> > > me
> > > > > > think
> > > > > > > > that
> > > > > > > > > is not the case ;). I let Kris to clarify that with you. My
> > > view
> > > > is
> > > > > > > that
> > > > > > > > > data-index is a way to query recently closed and active
> > > processes
> > > > > > (the
> > > > > > > > key
> > > > > > > > > here is the definition of recently, which in my opinion
> > should
> > > be
> > > > > > > > > configurable)
> > > > > > > > > But, besides that discussion and being pragmatic, keeping
> > > > finishing
> > > > > > > > process
> > > > > > > > > instances "for a while" in DataIndex was the only way for
> > users
> > > > to
> > > > > > > query
> > > > > > > > > the result of straight through processes. That's a function
> > > that
> > > > > > cannot
> > > > > > > > be
> > > > > > > > > removed right now
> > > > > > > > >
> > > > > > > > > On Mon, Feb 19, 2024 at 1:33 PM Alex Porcelli <
> > > > porcelli@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > if data index was supposed to provide snapshot view of
> the
> > > > > process
> > > > > > > > > > instance… why do we keep it after the process instance is
> > > > > finished?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Feb 19, 2024 at 7:12 AM Francisco Javier Tirado
> > > Sarti <
> > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Martin.
> > > > > > > > > > > After taking a deeper look at this, I realize that the
> > > > > behaviour
> > > > > > is
> > > > > > > > the
> > > > > > > > > > > expected one.
> > > > > > > > > > > Runtimes DB does not track the completed process
> instance
> > > > > (that's
> > > > > > > > what
> > > > > > > > > > the
> > > > > > > > > > > JDBCProcessInstances warn is telling us), but
> DataIndex,
> > as
> > > > > > > expected,
> > > > > > > > > is
> > > > > > > > > > > tracking it in processes and nodes table. And yes it
> will
> > > > grow
> > > > > > over
> > > > > > > > > time.
> > > > > > > > > > > What we need is some configurable purge mechanism for
> > > > > DataIndex,
> > > > > > so
> > > > > > > > it
> > > > > > > > > > > eventually removes older completed process instances.
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Feb 13, 2024 at 12:59 PM Francisco Javier
> Tirado
> > > > Sarti
> > > > > <
> > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Martin,
> > > > > > > > > > > > Good catch!. Looks like the skipping performed for
> > > process
> > > > > > > > instances
> > > > > > > > > is
> > > > > > > > > > > > not applied to node instances. Something we
> definitely
> > > need
> > > > > to
> > > > > > > > review
> > > > > > > > > > on
> > > > > > > > > > > > the runtimes side.
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Feb 12, 2024 at 11:59 PM Martin Weiler
> > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> On a somewhat related note, testing a simple
> workflow
> > > > (start
> > > > > > ->
> > > > > > > > > script
> > > > > > > > > > > >> node -> end), I see the following messages in the
> > logs:
> > > > > > > > > > > >> 2024-02-12 22:49:50,493 28758dde544c WARN
> > > > > > > > > > > >>
> > > [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> > > > > > > > > > > >> (executor-thread-3) Skipping create of process
> > instance
> > > > id:
> > > > > > > > > > > >> 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2
> > > > > > > > > > > >>
> > > > > > > > > > > >> So far, so good. And I'd expect to see no trace of
> > this
> > > > > > process
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > >> database if I don't have data audit enabled.
> > > > > > > > > > > >>
> > > > > > > > > > > >> However, the 'processes' table contains a row with
> > > > state=2,
> > > > > > with
> > > > > > > > > > related
> > > > > > > > > > > >> entries in the 'nodes' table. In a load test, I see
> > > these
> > > > > > tables
> > > > > > > > > grow
> > > > > > > > > > > >> significantly over time. Am I missing something to
> > have
> > > > > these
> > > > > > > > > entries
> > > > > > > > > > > >> cleaned up automatically?
> > > > > > > > > > > >>
> > > > > > > > > > > >> ________________________________________
> > > > > > > > > > > >> From: Martin Weiler <mw...@ibm.com.INVALID>
> > > > > > > > > > > >> Sent: Monday, February 12, 2024 3:40 PM
> > > > > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > > > > >> Subject: [EXTERNAL] RE: [DISCUSSION] Performance
> > issues
> > > > with
> > > > > > > > > > data-index
> > > > > > > > > > > >> persistence addon
> > > > > > > > > > > >>
> > > > > > > > > > > >> Thanks everyone for your input. Based on this
> > > discussion,
> > > > I
> > > > > > > opened
> > > > > > > > > the
> > > > > > > > > > > >> following PR:
> > > > > > > > > > > >>
> > > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1985
> > > > > > > > > > > >>
> > > > > > > > > > > >> With this change, the performance seems to be stable
> > > over
> > > > > > time:
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
> > > > > > > > > > > >>
> > > > > > > > > > > >> Martin
> > > > > > > > > > > >>
> > > > > > > > > > > >> ________________________________________
> > > > > > > > > > > >> From: Gonzalo Muñoz <gm...@apache.org>
> > > > > > > > > > > >> Sent: Friday, February 9, 2024 9:42 AM
> > > > > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > > > > >> Subject: [EXTERNAL] Re: [DISCUSSION] Performance
> > issues
> > > > with
> > > > > > > > > > data-index
> > > > > > > > > > > >> persistence addon
> > > > > > > > > > > >>
> > > > > > > > > > > >> Great work Francisco,
> > > > > > > > > > > >> Martin, take a look at this link with some related
> > tips
> > > > (in
> > > > > > case
> > > > > > > > you
> > > > > > > > > > > find
> > > > > > > > > > > >> it useful):
> > > > > > > > > > > >>
> > > > > > https://www.cybertec-postgresql.com/en/index-your-foreign-key/
> > > > > > > > > > > >>
> > > > > > > > > > > >> El vie, 9 feb 2024 a las 17:20, Francisco Javier
> > Tirado
> > > > > Sarti
> > > > > > (<
> > > > > > > > > > > >> ftirados@redhat.com>) escribió:
> > > > > > > > > > > >>
> > > > > > > > > > > >> > For the moment being, we will keep JPA till we
> > exhaust
> > > > all
> > > > > > > > > > > >> possibilities,
> > > > > > > > > > > >> > let's call switching from jpa to jdbc our hidden
> > plan
> > > B
> > > > ;)
> > > > > > > > > > > >> > I already told Martin, but in order everyone to
> > know,
> > > > just
> > > > > > > after
> > > > > > > > > > > writing
> > > > > > > > > > > >> > the previous email, I thought "what if Postgres is
> > not
> > > > > > > > > automatically
> > > > > > > > > > > >> > indexing foreign keys like mysql?" and, eureka
> > > > > > > > > > > >> > Postgres doc
> > > > > > > > > > > >> >
> > > > > > https://www.postgresql.org/docs/current/ddl-constraints.html
> > > > > > > > > > > >> > Mysql doc
> > > > > > > > > > > >> >
> > > > > > > > >
> > > > >
> https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> > > > > > > > > > > >> > These are the relevant excerpt
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > *Postgresql*
> > > > > > > > > > > >> > *A foreign key must reference columns that either
> > are
> > > a
> > > > > > > primary
> > > > > > > > > key
> > > > > > > > > > or
> > > > > > > > > > > >> form
> > > > > > > > > > > >> > a unique constraint, or are columns from a
> > non-partial
> > > > > > unique
> > > > > > > > > index.
> > > > > > > > > > > >> This
> > > > > > > > > > > >> > means that the referenced columns always have an
> > index
> > > > to
> > > > > > > allow
> > > > > > > > > > > >> efficient
> > > > > > > > > > > >> > lookups on whether a referencing row has a match.
> > > Since
> > > > a
> > > > > > > DELETE
> > > > > > > > > of
> > > > > > > > > > a
> > > > > > > > > > > >> row
> > > > > > > > > > > >> > from the referenced table or an UPDATE of a
> > referenced
> > > > > > column
> > > > > > > > will
> > > > > > > > > > > >> require
> > > > > > > > > > > >> > a scan of the referencing table for rows matching
> > the
> > > > old
> > > > > > > value,
> > > > > > > > > it
> > > > > > > > > > is
> > > > > > > > > > > >> > often a good idea to index the referencing columns
> > > too.
> > > > > > > Because
> > > > > > > > > this
> > > > > > > > > > > is
> > > > > > > > > > > >> not
> > > > > > > > > > > >> > always needed, and there are many choices
> available
> > on
> > > > how
> > > > > > to
> > > > > > > > > index,
> > > > > > > > > > > the
> > > > > > > > > > > >> > declaration of a foreign key constraint does not
> > > > > > automatically
> > > > > > > > > > create
> > > > > > > > > > > an
> > > > > > > > > > > >> > index on the referencing columns.*
> > > > > > > > > > > >> > *Mysql*
> > > > > > > > > > > >> > *MySQL requires that foreign key columns be
> indexed;
> > > if
> > > > > you
> > > > > > > > > create a
> > > > > > > > > > > >> table
> > > > > > > > > > > >> > with a foreign key constraint but no index on a
> > given
> > > > > > column,
> > > > > > > an
> > > > > > > > > > index
> > > > > > > > > > > >> is
> > > > > > > > > > > >> > created. *
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > So I asked Martin to especially create an index
> for
> > > > > > > > > > > process_instance_id
> > > > > > > > > > > >> > column on nodes table
> > > > > > > > > > > >> > I think that will fix the problem detected on the
> > > thread
> > > > > > dump.
> > > > > > > > > > > >> > The simpler process test to verify queries are
> fine
> > > > still
> > > > > > > > stands,
> > > > > > > > > > > >> though ;)
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <
> > > > > > > > tzimanyi@apache.org
> > > > > > > > > >
> > > > > > > > > > > >> wrote:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > > I always preferred pure JDBC over Hibernate
> > myself,
> > > > just
> > > > > > for
> > > > > > > > the
> > > > > > > > > > > sake
> > > > > > > > > > > >> of
> > > > > > > > > > > >> > > control of what is happening :) So I would not
> -1
> > > that
> > > > > > > myself.
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > Tibor
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado
> > > > Sarti <
> > > > > > > > > > > >> > > ftirados@redhat.com>
> > > > > > > > > > > >> > > napísal(a):
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > > Hi,
> > > > > > > > > > > >> > > > Usually I do not want to talk about work in
> > > progress
> > > > > > > because
> > > > > > > > > > > >> > preliminary
> > > > > > > > > > > >> > > > conclusions are pretty volatile but, well,
> there
> > > > are a
> > > > > > > > couple
> > > > > > > > > of
> > > > > > > > > > > >> things
> > > > > > > > > > > >> > > > that can be concluded from the really valuable
> > > > > > information
> > > > > > > > > that
> > > > > > > > > > > >> Martin
> > > > > > > > > > > >> > > > provided.
> > > > > > > > > > > >> > > > 1) In order to be able to determine if the
> > number
> > > of
> > > > > > > > > statements
> > > > > > > > > > is
> > > > > > > > > > > >> > larger
> > > > > > > > > > > >> > > > than expected, I asked Martin to test with a
> > > simpler
> > > > > > > process
> > > > > > > > > > > >> > definition.
> > > > > > > > > > > >> > > > One with just three nodes: start, script and
> > end.
> > > > The
> > > > > > > script
> > > > > > > > > one
> > > > > > > > > > > >> should
> > > > > > > > > > > >> > > > change just one variable. This way we can
> > analyze
> > > if
> > > > > the
> > > > > > > > > number
> > > > > > > > > > of
> > > > > > > > > > > >> > > queries
> > > > > > > > > > > >> > > > is the expected one. From the single log
> (audit
> > > was
> > > > > > > > activated
> > > > > > > > > > > them)
> > > > > > > > > > > >> my
> > > > > > > > > > > >> > > > conclusion is that the number of
> insert/updates
> > > over
> > > > > > > > processes
> > > > > > > > > > and
> > > > > > > > > > > >> > nodes
> > > > > > > > > > > >> > > > (there a lot over task, that I will prefer to
> > skip
> > > > for
> > > > > > > now,
> > > > > > > > > baby
> > > > > > > > > > > >> steps)
> > > > > > > > > > > >> > > is
> > > > > > > > > > > >> > > > the expected one.
> > > > > > > > > > > >> > > > 2) Analysing the thread dump, we see around 15
> > > > threads
> > > > > > > > > executing
> > > > > > > > > > > >> this
> > > > > > > > > > > >> > > line
> > > > > > > > > > > >> > > > at
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > > > > > > > > > >> > > > so its pretty clear the code to be optimized
> ;).
> > > I'm
> > > > > > > > > evaluating
> > > > > > > > > > > >> > > > possibilities within JPA/Hibernate, but I'm
> > > starting
> > > > > to
> > > > > > > > think
> > > > > > > > > > that
> > > > > > > > > > > >> it
> > > > > > > > > > > >> > > might
> > > > > > > > > > > >> > > > be better to switch to JDBC and skip
> hibernate.
> > > Our
> > > > > > lives
> > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > >> > > simpler,
> > > > > > > > > > > >> > > > especially with a schema relatively simple
> like
> > > ours
> > > > > > (that
> > > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > >> my
> > > > > > > > > > > >> > > > recommendation if I was an external
> consultant)
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <
> > > > > > > > > > tzimanyi@apache.org
> > > > > > > > > > > >
> > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > > Hi,
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > this will be a bit off-topic. However as far
> > as
> > > > > > > > > performance, I
> > > > > > > > > > > >> think
> > > > > > > > > > > >> > we
> > > > > > > > > > > >> > > > > should think about that we have string
> primary
> > > > keys
> > > > > > > > (IDs). I
> > > > > > > > > > > would
> > > > > > > > > > > >> > > expect
> > > > > > > > > > > >> > > > > the database systems are much better with
> > > indexing
> > > > > > > numeric
> > > > > > > > > > keys
> > > > > > > > > > > >> than
> > > > > > > > > > > >> > > > > strings. I remember from the past, when I
> was
> > > > > working
> > > > > > > with
> > > > > > > > > > DBs,
> > > > > > > > > > > >> that
> > > > > > > > > > > >> > > > using
> > > > > > > > > > > >> > > > > strings as keys or indexes was a discouraged
> > > > > practice.
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > Best regards,
> > > > > > > > > > > >> > > > > Tibor
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler
> > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > >
> > > > > > > > > > > >> > > > > napísal(a):
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > > I changed the test to use MongoDB [1] and
> I
> > > > don't
> > > > > > see
> > > > > > > a
> > > > > > > > > > > >> performance
> > > > > > > > > > > >> > > > > > degradation with this setup [2].
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > Please keep us posted of your findings.
> > > Thanks!
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > Martin
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > [1]
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> >
> > > > > > > > > > >
> > > > > > > >
> > > > >
> > https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > > > > > > > > >> > > > > > [2]
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > ________________________________________
> > > > > > > > > > > >> > > > > > From: Francisco Javier Tirado Sarti <
> > > > > > > > ftirados@redhat.com>
> > > > > > > > > > > >> > > > > > Sent: Wednesday, February 7, 2024 11:40 AM
> > > > > > > > > > > >> > > > > > To: dev@kie.apache.org
> > > > > > > > > > > >> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION]
> > > Performance
> > > > > > > issues
> > > > > > > > > with
> > > > > > > > > > > >> > > data-index
> > > > > > > > > > > >> > > > > > persistence addon
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > yes, it can be index degradation because
> of
> > > > size,
> > > > > > but
> > > > > > > I
> > > > > > > > > > > believe
> > > > > > > > > > > >> (I
> > > > > > > > > > > >> > > > might
> > > > > > > > > > > >> > > > > be
> > > > > > > > > > > >> > > > > > wrong) the db is too small (yet) for that.
> > > > > > > > > > > >> > > > > > But, eventually, Postgres, when the DB is
> > huge
> > > > > > enough,
> > > > > > > > > > > >> unavoidably
> > > > > > > > > > > >> > > will
> > > > > > > > > > > >> > > > > > behave like the graphic that Martin sent.
> > > > > > > > > > > >> > > > > > Since I believe we are not huge enough
> > (yet),
> > > > lets
> > > > > > > rule
> > > > > > > > > out
> > > > > > > > > > > >> another
> > > > > > > > > > > >> > > > issue
> > > > > > > > > > > >> > > > > > by analysing the sql logs (I requested
> those
> > > to
> > > > > > Martin
> > > > > > > > > > offline
> > > > > > > > > > > >> and
> > > > > > > > > > > >> > he
> > > > > > > > > > > >> > > > is
> > > > > > > > > > > >> > > > > > going to kindly collect them).
> > > > > > > > > > > >> > > > > > Also Im curious to know if Mongo behave in
> > the
> > > > > same
> > > > > > > way.
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique
> > > Gonzalez
> > > > > > > > Martinez <
> > > > > > > > > > > >> > > > > > egonzalez@apache.org> wrote:
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > > Hi Francisco,
> > > > > > > > > > > >> > > > > > > I would highly recommend to check
> indexes
> > > and
> > > > > how
> > > > > > > the
> > > > > > > > > > > updates
> > > > > > > > > > > >> > work
> > > > > > > > > > > >> > > in
> > > > > > > > > > > >> > > > > > data
> > > > > > > > > > > >> > > > > > > index to avoid full scan table and lock
> > the
> > > > full
> > > > > > > > table.
> > > > > > > > > > Some
> > > > > > > > > > > >> db
> > > > > > > > > > > >> > are
> > > > > > > > > > > >> > > > > very
> > > > > > > > > > > >> > > > > > > sensitive to that.
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > El mié, 7 feb 2024, 18:41, Francisco
> > Javier
> > > > > Tirado
> > > > > > > > > Sarti <
> > > > > > > > > > > >> > > > > > > ftirados@redhat.com> escribió:
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > > Hi Martin,
> > > > > > > > > > > >> > > > > > > > While I analyze the data, let me ask
> you
> > > if
> > > > it
> > > > > > is
> > > > > > > > > > possible
> > > > > > > > > > > >> to
> > > > > > > > > > > >> > > > perform
> > > > > > > > > > > >> > > > > > > > another check (similar in a way to
> > > disabling
> > > > > > > > > data-index
> > > > > > > > > > > like
> > > > > > > > > > > >> > you
> > > > > > > > > > > >> > > > do)
> > > > > > > > > > > >> > > > > > Can
> > > > > > > > > > > >> > > > > > > > you switch to MongoDB persistence and
> > > check
> > > > if
> > > > > > the
> > > > > > > > > same
> > > > > > > > > > > >> > > degradation
> > > > > > > > > > > >> > > > > > that
> > > > > > > > > > > >> > > > > > > is
> > > > > > > > > > > >> > > > > > > > there for postgres remains?
> > > > > > > > > > > >> > > > > > > > I do not know if this is feasible but
> > will
> > > > > > > certainly
> > > > > > > > > > > >> indicate
> > > > > > > > > > > >> > the
> > > > > > > > > > > >> > > > > > problem
> > > > > > > > > > > >> > > > > > > > is on the postgres storage layer and I
> > do
> > > > not
> > > > > > > have a
> > > > > > > > > > clear
> > > > > > > > > > > >> > > > prediction
> > > > > > > > > > > >> > > > > > of
> > > > > > > > > > > >> > > > > > > > what we will see when doing this
> switch.
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin
> > > Weiler
> > > > > > > > > > > >> > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > > > wrote:
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > > > Hi Francisco,
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > thanks for your work on this
> important
> > > > > topic!
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > I would like to share some test
> > results
> > > > > here,
> > > > > > > > which
> > > > > > > > > > > might
> > > > > > > > > > > >> > help
> > > > > > > > > > > >> > > to
> > > > > > > > > > > >> > > > > > > improve
> > > > > > > > > > > >> > > > > > > > > the codebase even further. I am
> using
> > > the
> > > > > > jmeter
> > > > > > > > > based
> > > > > > > > > > > >> test
> > > > > > > > > > > >> > > case
> > > > > > > > > > > >> > > > > from
> > > > > > > > > > > >> > > > > > > > Pere
> > > > > > > > > > > >> > > > > > > > > and Enrique (thanks guys!) [1] which
> > > uses
> > > > a
> > > > > > load
> > > > > > > > of
> > > > > > > > > 30
> > > > > > > > > > > >> > threads
> > > > > > > > > > > >> > > to
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > 1) start a new process instance
> (POST)
> > > > > > > > > > > >> > > > > > > > > 2) retrieve tasks for a user (GET)
> > > > > > > > > > > >> > > > > > > > > 3) fetches task details (GET)
> > > > > > > > > > > >> > > > > > > > > 4) complete a task (POST)
> > > > > > > > > > > >> > > > > > > > > 5) execute a query on data-audit
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > With this test setup, I noticed that
> > the
> > > > > > > > performance
> > > > > > > > > > for
> > > > > > > > > > > >> the
> > > > > > > > > > > >> > > POST
> > > > > > > > > > > >> > > > > > > > > requests, in particular the one to
> > > start a
> > > > > new
> > > > > > > > > process
> > > > > > > > > > > >> > > instance,
> > > > > > > > > > > >> > > > > > > degrades
> > > > > > > > > > > >> > > > > > > > > over time - see graph [2]. If I run
> > the
> > > > same
> > > > > > > test
> > > > > > > > > > > without
> > > > > > > > > > > >> > > > > data-index,
> > > > > > > > > > > >> > > > > > > > then
> > > > > > > > > > > >> > > > > > > > > there is no such performance
> > degradation
> > > > > [3].
> > > > > > > You
> > > > > > > > > can
> > > > > > > > > > > >> find a
> > > > > > > > > > > >> > > > thread
> > > > > > > > > > > >> > > > > > > dump
> > > > > > > > > > > >> > > > > > > > > captured a few minutes into the
> first
> > > test
> > > > > > here
> > > > > > > > [4]
> > > > > > > > > > that
> > > > > > > > > > > >> > might
> > > > > > > > > > > >> > > > help
> > > > > > > > > > > >> > > > > > to
> > > > > > > > > > > >> > > > > > > > see
> > > > > > > > > > > >> > > > > > > > > some of the contention points.
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > I'd appreciate if you could take a
> > look
> > > > and
> > > > > > see
> > > > > > > if
> > > > > > > > > > there
> > > > > > > > > > > >> is
> > > > > > > > > > > >> > > > > something
> > > > > > > > > > > >> > > > > > > > that
> > > > > > > > > > > >> > > > > > > > > can be further improved based on
> your
> > > > > previous
> > > > > > > > work.
> > > > > > > > > > If
> > > > > > > > > > > >> you
> > > > > > > > > > > >> > > need
> > > > > > > > > > > >> > > > > any
> > > > > > > > > > > >> > > > > > > > > additional data, let me know, but
> > > > otherwise
> > > > > it
> > > > > > > is
> > > > > > > > > > > >> > > straightforward
> > > > > > > > > > > >> > > > > to
> > > > > > > > > > > >> > > > > > > run
> > > > > > > > > > > >> > > > > > > > > the jmeter test as well.
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > Thanks,
> > > > > > > > > > > >> > > > > > > > > Martin
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > [1]
> > > > > > > > > > > >>
> > https://github.com/pefernan/job-service-refactor-test/
> > > > > > > > > > > >> > > > > > > > > [2]
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > > > > > > > >> > > > > > > > > [3]
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > > > > > > > >> > > > > > > > > [4]
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > >
> > ________________________________________
> > > > > > > > > > > >> > > > > > > > > From: Francisco Javier Tirado Sarti
> <
> > > > > > > > > > > ftirados@redhat.com>
> > > > > > > > > > > >> > > > > > > > > Sent: Wednesday, January 17, 2024
> 9:13
> > > AM
> > > > > > > > > > > >> > > > > > > > > To: dev@kie.apache.org
> > > > > > > > > > > >> > > > > > > > > Cc: Pere Fernandez Perez
> > > > > > > > > > > >> > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION]
> > > > > > Performance
> > > > > > > > > > issues
> > > > > > > > > > > >> with
> > > > > > > > > > > >> > > > > > data-index
> > > > > > > > > > > >> > > > > > > > > persistence addon
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > Hi Alex,
> > > > > > > > > > > >> > > > > > > > > I did not take times (which depends
> > on a
> > > > > > number
> > > > > > > of
> > > > > > > > > > > >> variables
> > > > > > > > > > > >> > > that
> > > > > > > > > > > >> > > > > > > > > drastically change between
> > > environments),
> > > > > but
> > > > > > > > verify
> > > > > > > > > > > that
> > > > > > > > > > > >> the
> > > > > > > > > > > >> > > > > number
> > > > > > > > > > > >> > > > > > of
> > > > > > > > > > > >> > > > > > > > > updates has been reduced drastically
> > > > without
> > > > > > > > losing
> > > > > > > > > > > >> > > > functionality,
> > > > > > > > > > > >> > > > > > > which
> > > > > > > > > > > >> > > > > > > > is
> > > > > > > > > > > >> > > > > > > > > objectively a good thing. If before
> > the
> > > > > > change,
> > > > > > > > for
> > > > > > > > > > > every
> > > > > > > > > > > >> > node
> > > > > > > > > > > >> > > > > > > executed,
> > > > > > > > > > > >> > > > > > > > we
> > > > > > > > > > > >> > > > > > > > > have an update for every node
> > previously
> > > > > > > executed,
> > > > > > > > > so
> > > > > > > > > > > if a
> > > > > > > > > > > >> > > > process
> > > > > > > > > > > >> > > > > > have
> > > > > > > > > > > >> > > > > > > > 50
> > > > > > > > > > > >> > > > > > > > > nodes to execute, we were performing
> > > > nearly
> > > > > > > > 50*51/2
> > > > > > > > > > > >> updates,
> > > > > > > > > > > >> > > > which
> > > > > > > > > > > >> > > > > > > gives
> > > > > > > > > > > >> > > > > > > > us
> > > > > > > > > > > >> > > > > > > > > a total of  1275 updates, now we
> have
> > > just
> > > > > one
> > > > > > > for
> > > > > > > > > > every
> > > > > > > > > > > >> node
> > > > > > > > > > > >> > > > being
> > > > > > > > > > > >> > > > > > > > > executed, implying a total of 50
> > > updates.
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex
> > > > > Porcelli
> > > > > > <
> > > > > > > > > > > >> > > alex@porcelli.me>
> > > > > > > > > > > >> > > > > > > wrote:
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > Francisco,
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > I noticed that your PR has been
> > > merged,
> > > > > but
> > > > > > I
> > > > > > > > was
> > > > > > > > > > > >> expecting
> > > > > > > > > > > >> > > (at
> > > > > > > > > > > >> > > > > > least
> > > > > > > > > > > >> > > > > > > > > > was my understanding from this
> > thread)
> > > > > that
> > > > > > > > before
> > > > > > > > > > > >> merging
> > > > > > > > > > > >> > > some
> > > > > > > > > > > >> > > > > > > > > > benchmark data would be shared in
> > > > advance
> > > > > -
> > > > > > to
> > > > > > > > > > assess
> > > > > > > > > > > >> the
> > > > > > > > > > > >> > > > > > > cost/benefit
> > > > > > > > > > > >> > > > > > > > > > of such a decent size change.
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > Do you have any information to
> > share?
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM
> > > > Francisco
> > > > > > > Javier
> > > > > > > > > > > Tirado
> > > > > > > > > > > >> > Sarti
> > > > > > > > > > > >> > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > Yes, as intended, now we have
> one
> > > > select
> > > > > > and
> > > > > > > > one
> > > > > > > > > > > >> > > > insert/update
> > > > > > > > > > > >> > > > > > per
> > > > > > > > > > > >> > > > > > > > node
> > > > > > > > > > > >> > > > > > > > > > > event.
> > > > > > > > > > > >> > > > > > > > > > > I moved the PR as ready for
> review
> > > and
> > > > > > give
> > > > > > > > > @Pere
> > > > > > > > > > > >> > Fernandez
> > > > > > > > > > > >> > > > > Perez
> > > > > > > > > > > >> > > > > > > > > > > <pe...@redhat.com>
> permission
> > to
> > > > the
> > > > > > > > branch
> > > > > > > > > so
> > > > > > > > > > > he
> > > > > > > > > > > >> can
> > > > > > > > > > > >> > > > edit
> > > > > > > > > > > >> > > > > it
> > > > > > > > > > > >> > > > > > > in
> > > > > > > > > > > >> > > > > > > > > the
> > > > > > > > > > > >> > > > > > > > > > > next two weeks (Ill be on PTO)
> if
> > > > > > desired,
> > > > > > > > > before
> > > > > > > > > > > >> > merging.
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM
> > Alex
> > > > > > > Porcelli
> > > > > > > > <
> > > > > > > > > > > >> > > > > alex@porcelli.me>
> > > > > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > Cool, thank you Francisco!
> > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > Did you manage to get some
> > > > preliminary
> > > > > > > data
> > > > > > > > > > about
> > > > > > > > > > > >> > > > > improvements?
> > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > On Thu, Dec 21, 2023 at
> 11:52 AM
> > > > > > Francisco
> > > > > > > > > > Javier
> > > > > > > > > > > >> > Tirado
> > > > > > > > > > > >> > > > > Sarti
> > > > > > > > > > > >> > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > Yes, after some delay
> because
> > of
> > > > > > > quarkus 3
> > > > > > > > > > > >> migration.
> > > > > > > > > > > >> > > Im
> > > > > > > > > > > >> > > > > > > refining
> > > > > > > > > > > >> > > > > > > > > > this
> > > > > > > > > > > >> > > > > > > > > > > > > draft PR
> > > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > >
> > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > On Thu, Dec 21, 2023 at
> > 5:48 PM
> > > > Alex
> > > > > > > > > Porcelli
> > > > > > > > > > <
> > > > > > > > > > > >> > > > > > > alex@porcelli.me>
> > > > > > > > > > > >> > > > > > > > > > wrote:
> > > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > Any update or new findings
> > on
> > > > this
> > > > > > > > topic?
> > > > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > On Tue, Nov 28, 2023 at
> > > 8:38 AM
> > > > > > > > Francisco
> > > > > > > > > > > Javier
> > > > > > > > > > > >> > > Tirado
> > > > > > > > > > > >> > > > > > Sarti
> > > > > > > > > > > >> > > > > > > > > > > > > > <ft...@redhat.com>
> > wrote:
> > > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > >> > > > > > > > > > > > > > > After considering
> > different
> > > > > > options
> > > > > > > to
> > > > > > > > > > > improve
> > > > > > > > > > > >> > > > > > performance,
> > > > > > > > > > > >> > > > > > > > we
> > > > > > > > > > > >> > > > > > > > > > feel
> > > > > > > > > > > >> > > > > > > > > > > > that
> > > > > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > > > > > >> > > > > > > > > > > > > > > is time to "partially"
> > move
> > > > away
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > > > >> current
> > > > > > > > > > > >> > > Map
> > > > > > > > > > > >> > > > > > style
> > > > > > > > > > > >> > > > > > > > > > > > interface (
> > > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > > > > > > >> > > > > > > > > > > > > > )
> > > > > > > > > > > >> > > > > > > > > > > > > > > which was shared with
> > > Trusty,
> > > > to
> > > > > > one
> > > > > > > > > more
> > > > > > > > > > > >> > suitable
> > > > > > > > > > > >> > > > for
> > > > > > > > > > > >> > > > > > > usage
> > > > > > > > > > > >> > > > > > > > > > with a
> > > > > > > > > > > >> > > > > > > > > > > > > > > relational DB like
> > > postgresql
> > > > > (but
> > > > > > > > still
> > > > > > > > > > > >> > compatible
> > > > > > > > > > > >> > > > > with
> > > > > > > > > > > >> > > > > > > big
> > > > > > > > > > > >> > > > > > > > > > table
> > > > > > > > > > > >> > > > > > > > > > > > dbs).
> > > > > > > > > > > >> > > > > > > > > > > > > > > The idea will be to
> > replace
> > > > > > generic
> > > > > > > > > > Storage
> > > > > > > > > > > >> > > interface
> > > > > > > > > > > >> > > > > by
> > > > > > > > > > > >> > > > > > > four
> > > > > > > > > > > >> > > > > > > > > > > > specific
> > > > > > > > > > > >> > > > > > > > > > > > > > > interfaces (which will
> > > inherit
> > > > > > from
> > > > > > > a
> > > > > > > > > > common
> > > > > > > > > > > >> one
> > > > > > > > > > > >> > > that
> > > > > > > > > > > >> > > > > > keeps
> > > > > > > > > > > >> > > > > > > > the
> > > > > > > > > > > >> > > > > > > > > > query
> > > > > > > > > > > >> > > > > > > > > > > > > > part
> > > > > > > > > > > >> > > > > > > > > > > > > > > at is it. with get and
> > query
> > > > > > > methods),
> > > > > > > > > > that
> > > > > > > > > > > >> will
> > > > > > > > > > > >> > > > > include
> > > > > > > > > > > >> > > > > > > the
> > > > > > > > > > > >> > > > > > > > > > required
> > > > > > > > > > > >> > > > > > > > > > > > > > > modification operations
> > for
> > > > the
> > > > > > four
> > > > > > > > > > > DataIndex
> > > > > > > > > > > >> > > > > "domains":
> > > > > > > > > > > >> > > > > > > > > > > > > > processinstance,
> > > > > > > > > > > >> > > > > > > > > > > > > > > usertask,
> > processdefinitions
> > > > and
> > > > > > > jobs.
> > > > > > > > > > Those
> > > > > > > > > > > >> > > > interfaces
> > > > > > > > > > > >> > > > > > > will
> > > > > > > > > > > >> > > > > > > > > > define
> > > > > > > > > > > >> > > > > > > > > > > > > > methods
> > > > > > > > > > > >> > > > > > > > > > > > > > > like addNode,
> addVariable,
> > > > > > > updateTask,
> > > > > > > > > > > >> > > > > addAttachment.....
> > > > > > > > > > > >> > > > > > > > that
> > > > > > > > > > > >> > > > > > > > > > will
> > > > > > > > > > > >> > > > > > > > > > > > allow
> > > > > > > > > > > >> > > > > > > > > > > > > > > the persistent layer
> > > > > > implementation
> > > > > > > > to
> > > > > > > > > > just
> > > > > > > > > > > >> > update
> > > > > > > > > > > >> > > > the
> > > > > > > > > > > >> > > > > > > > needed
> > > > > > > > > > > >> > > > > > > > > > info
> > > > > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > > > >> > > > > > > > > > > > > > > DB  (for example, for
> > > addNode
> > > > in
> > > > > > > > > Postgres,
> > > > > > > > > > > >> just
> > > > > > > > > > > >> > > > insert
> > > > > > > > > > > >> > > > > a
> > > > > > > > > > > >> > > > > > > row
> > > > > > > > > > > >> > > > > > > > > into
> > > > > > > > > > > >> > > > > > > > > > > > nodes
> > > > > > > > > > > >> > > > > > > > > > > > > > > table, for addNode in
> > Mongo,
> > > > > > > basically
> > > > > > > > > the
> > > > > > > > > > > >> same
> > > > > > > > > > > >> > > > atomic
> > > > > > > > > > > >> > > > > > > upsert
> > > > > > > > > > > >> > > > > > > > > > > > operation
> > > > > > > > > > > >> > > > > > > > > > > > > > > that is currently done).
> > > > > > Therefore,
> > > > > > > we
> > > > > > > > > > > >> increase
> > > > > > > > > > > >> > > > > > performance
> > > > > > > > > > > >> > > > > > > > for
> > > > > > > > > > > >> > > > > > > > > > > > Postgres
> > > > > > > > > > > >> > > > > > > > > > > > > > > and keep the current one
> > for
> > > > > > Mongo.
> > > > > > > > The
> > > > > > > > > > > >> current
> > > > > > > > > > > >> > DB
> > > > > > > > > > > >> > > > > > schemas
> > > > > > > > > > > >> > > > > > > > > won't
> > > > > > > > > > > >> > > > > > > > > > be
> > > > > > > > > > > >> > > > > > > > > > > > > > > touched.
> > > > > > > > > > > >> > > > > > > > > > > > > > > Since the code change is
> > > > large,
> > > > > I
> > > > > > do
> > > > > > > > not
> > > > > > > > > > > think
> > > > > > > > > > > >> > I'll
> > > > > > > > > > > >> > > > be
> > > > > > > > > > > >> > > > > > able
> > > > > > > > > > > >> > > > > > > > to
> > > > > > > > > > > >> > > > > > > > > > have
> > > > > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > > > >> > > > > > > > > > > > > > PR
> > > > > > > > > > > >> > > > > > > > > > > > > > > ready till next week.
> > > > > > > > > > > >> > > > > > > > > > > > > > > But before starting,
> > please
> > > > let
> > > > > me
> > > > > > > > know
> > > > > > > > > if
> > > > > > > > > > > >> that
> > > > > > > > > > > >> > > > > approach
> > > > > > > > > > > >> > > > > > is
> > > > > > > > > > > >> > > > > > > > > fine
> > > > > > > > > > > >> > > > > > > > > > for
> > > > > > > > > > > >> > > > > > > > > > > > you.
> > > > > > > > > > > >> > > > > > > > > > > > > > > Best regards.
> > > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> > > > 6:55 PM
> > > > > > Alex
> > > > > > > > > > > Porcelli
> > > > > > > > > > > >> <
> > > > > > > > > > > >> > > > > > > > > alex@porcelli.me>
> > > > > > > > > > > >> > > > > > > > > > > > wrote:
> > > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > > > Thank you Francisco to
> > > > getting
> > > > > > > > deeper
> > > > > > > > > on
> > > > > > > > > > > >> this…
> > > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > > > Looking forward to see
> > the
> > > > > > results
> > > > > > > > of
> > > > > > > > > > your
> > > > > > > > > > > >> > > > suggested
> > > > > > > > > > > >> > > > > > > > > > improvements.
> > > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023
> at
> > > > > 9:40 AM
> > > > > > > > > > Francisco
> > > > > > > > > > > >> > Javier
> > > > > > > > > > > >> > > > > Tirado
> > > > > > > > > > > >> > > > > > > > > Sarti <
> > > > > > > > > > > >> > > > > > > > > > > > > > > > ftirados@redhat.com>
> > > wrote:
> > > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > > > > I forgot to attach
> the
> > > > > queries
> > > > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > > > > On Fri, Nov 24, 2023
> > at
> > > > > > 3:04 PM
> > > > > > > > > > > Francisco
> > > > > > > > > > > >> > > Javier
> > > > > > > > > > > >> > > > > > Tirado
> > > > > > > > > > > >> > > > > > > > > > Sarti <
> > > > > > > > > > > >> > > > > > > > > > > > > > > > > ftirados@redhat.com
> >
> > > > wrote:
> > > > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > > > >> Hi,
> > > > > > > > > > > >> > > > > > > > > > > > > > > > >> A brief update on
> > this
> > > > > topic.
> > > > > > > > > > > >> > > > > > > > > > > > > > > > >> After doing a
> simple
> > > test
> > > > > > with
> > > > > > > > > > example
> > > > > > > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > > > > > >> > > > > > > > > > > > > > > > ,
> > > > > > > > > > > >> > > > > > > > > > > > > > > > >> the number of
> updates
> > > > over
> > > > > > > Nodes
> > > > > > > > > > table
> > > > > > > > > > > is
> > > > > > > > > > > >> > n*n,
> > > > > > > > > > > >> > > > so
> > > > > > > > > > > >> > > > > we
> > > > > > > > > > > >> > > > > > > > > manage
> > > > > > > > > > > >> > > > > > > > > > to
> > > > > > > > > > > >> > > > > > > > > > > > > > obtain a
> > > > > > > > > > > >> > > > > > > > > > > > > > > > >> perfect quadratic
> > > > > performance
> > > > > > > > > > > >> degradation.
> > > > > > > > > > > >> > The
> > > > > > > > > > > >> > > > > > problem
> > > > > > > > > > > >> > > > > > > > is
> > > > > > > > > > > >> > > > > > > > > > worse
> > > > > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > > > >> > > > > > > > > > > > > > > > case
> > > > >
>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Alex Porcelli <po...@apache.org>]

+1 for provide consistent behavior across the board

my 2c:

+1 for REST interface (preferred for external general management
perspective)
-1 for properties to auto clean; this can be achieved with cron and rest
call

Now about GraphQL: the platform today is heavily invested in such
interface, so it might make sense. BUT I’d consider it in a second moment.


On Wed, Feb 21, 2024 at 6:52 AM Francisco Javier Tirado Sarti <
ftirados@redhat.com> wrote:

> Hi Enrique,
> If we configure such policies through properties, it won't be enough to
> define a naming convention and rely on the target platform configuration
> capabilities (SpringBoot or Quarkus)?.
>
> On Wed, Feb 21, 2024 at 12:36 PM Enrique Gonzalez Martinez <
> egonzalez@apache.org> wrote:
>
> > Hi Francisco
> >
> > The discussion we need to have before how to achieve certain features, is
> > the overall of the user experience. If you want to clean it up that way
> it
> > is fine by me. I have nothing against setting policy related to some sort
> > of clean up for a process completed some time ago.
> >
> > As we discussed previously the data index is a snapshot of the last state
> > of the process instance included completed, but nothing was said once
> > completed when we need to clean that up so any policy is welcome. How to
> > achieve that policy is something completely different.
> >
> > The main problem is how every microservice is exposing that API to the
> end
> > user or consumer (other system) and which system is going to be the
> façade
> > for complex deployments and operations. That is the discussion I was
> > mentioning before.
> >
> > What I want to avoid is to set different policies among components and we
> > should strive to be as much as possible to offer certain capabilities in
> > the same fashion, e.g: clean up mechanism.
> > In the same way I want to avoid making the current system more complex
> than
> > they are. So far they are aligned offering one simple responsibility but
> I
> > would be against mixing for instance job service with data index.
> >
> > El mié, 21 feb 2024 a las 12:13, Francisco Javier Tirado Sarti (<
> > ftirados@redhat.com>) escribió:
> >
> > > Hi Enrique,
> > > In the case of data index I think the data to be purged is finished
> > process
> > > instances (I do not think we should remove process instance that has
> been
> > > alive for ages, even if it is very likely they are not going to be ever
> > > completed)
> > > Once you delete those process instances, you also delete the associated
> > > user tasks and jobs.
> > > Therefore the problem is relatively simple, to be able to configure how
> > > much time a completed process instance should remain in the data index
> > > database. We can take a simple approach: a property with a min duration
> > > that cannot be changed once data index is started; a slightly complex
> > one:
> > > the same property but watching it to react for changes; or the full
> > suite:
> > > an admin API to be able to change the policy at any moment.
> > > I think this discussion is worth having.
> > >
> > > On Wed, Feb 21, 2024 at 6:15 AM Enrique Gonzalez Martinez <
> > > egonzalez@apache.org> wrote:
> > >
> > > > Hi Martin, the main problem regarding the purge is because it is
> still
> > > > unclear the policy for what tech to use and the future components.
> > > >
> > > > Recently we had a discussion about proposing graphql for this sort of
> > > admin
> > > > tasks. So far for subsystems we have been using rest endpoints (like
> > > update
> > > > timers or modify human task or change processes). There is one
> > exception
> > > > which is the gateway that is pure graphql and somehow uses graphql
> for
> > > > everything making complex operations under the hood.
> > > >
> > > > This has somehow frozen the purge for data audit for a bit and the
> > > proposal
> > > > was to use rest endpoints to do the clean up in the component and
> offer
> > > the
> > > > graphql counterpart in the gateway promoting it to a first class
> > citizen
> > > > component instead of having it embedded in the data index.
> > > >
> > > > I would suggest to come up at least with a policy first regarding the
> > > > convention every component should address this.
> > > >
> > > >
> > > > El mar, 20 feb 2024, 23:39, Martin Weiler <mw...@ibm.com.invalid>
> > > > escribió:
> > > >
> > > > > IMO, it is good to have this discussion around data sanity now
> > instead
> > > of
> > > > > putting it off until later when data has already accumulated in
> > > > production
> > > > > environments.
> > > > >
> > > > > Based on the input here, we are dealing with three types of data:
> > > > > 1. Runtime data - active instances only, engine cleans up the data
> > > > > automatically at process instance end
> > > > > 2. Historic log data - data created by data-audit intended for long
> > > term
> > > > > storage
> > > > > 3. Data-index data - somehow this data falls in between the two
> > > > > aforementioned categories, with the idea of the data being
> "recent",
> > > but
> > > > > not restricted to active instances only
> > > > >
> > > > > We'd need purge strategies for both #2 and #3 (perhaps different
> > ones,
> > > or
> > > > > with different config settings) in order to prevent unlimited data
> > > > growth.
> > > > >
> > > > > ________________________________________
> > > > > From: Enrique Gonzalez Martinez <eg...@apache.org>
> > > > > Sent: Monday, February 19, 2024 7:11 AM
> > > > > To: dev@kie.apache.org
> > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > data-index
> > > > > persistence addon
> > > > >
> > > > > Hi Francisco,
> > > > > To give you more context about this.
> > > > >
> > > > > STP is a concept, a process with certain constraints: no
> persistence
> > > and
> > > > > returning the outcome in the call (sync execution with no idle
> > states).
> > > > It
> > > > > was a requirement from a user in the past. One of the requirements
> > was
> > > > > leaving no trail. In v7 was easy because you could disable the
> audit
> > in
> > > > > that case. Actually we have the same way to do what we did in v7 in
> > > here
> > > > as
> > > > > you can add/remove index just removing deps.
> > > > >
> > > > > We have the same outcome with different approaches and STP is
> already
> > > > > delivered.
> > > > >
> > > > > El lun, 19 feb 2024 a las 14:46, Francisco Javier Tirado Sarti (<
> > > > > ftirados@redhat.com>) escribió:
> > > > >
> > > > > > Regarding STP (which is not a concept that we have in the code. I
> > > mean
> > > > > STP
> > > > > > are processes as nonSTP are), I guess, as all processes, they
> were
> > > kept
> > > > > in
> > > > > > DataIndex once completed because users wanted (and still wants)
> to
> > > > check
> > > > > > the result once the call had been performed. If we want to leave
> no
> > > > trace
> > > > > > of them in DataIndex for some reason, we will need to make it a
> > > > > > Runtimes concept so DataIndex can handle them in a different way.
> > > > > >
> > > > > > On Mon, Feb 19, 2024 at 2:27 PM Enrique Gonzalez Martinez <
> > > > > > egonzalez@apache.org> wrote:
> > > > > >
> > > > > > > Alex:
> > > > > > > Right now the data index is working in the same way as it did
> in
> > v7
> > > > > with
> > > > > > > the emitters. The only difference between two impl is that in
> > here
> > > > the
> > > > > > > storage is pgsql instead elastic search.  You are right
> regarding
> > > is
> > > > a
> > > > > > > snapshot of the last state of the process but we did never
> define
> > > how
> > > > > > long
> > > > > > > would be alive that dats Honestly i am happy right now with the
> > way
> > > > it
> > > > > > > works. The clean up mechanism is still tbd because we still
> need
> > to
> > > > > > discuss
> > > > > > > other stuff first.
> > > > > > >
> > > > > > >
> > > > > > > Regarding stp is to leave no trail because u can get the
> outcome
> > > > > directly
> > > > > > > from the call. It was defined like that in v7. So there is no
> use
> > > for
> > > > > the
> > > > > > > index or the audit.
> > > > > > >
> > > > > > > El lun, 19 feb 2024, 14:13, Francisco Javier Tirado Sarti <
> > > > > > > ftirados@redhat.com> escribió:
> > > > > > >
> > > > > > > > Hi Alex,
> > > > > > > > There has been some confusion about the purpose of DataIndex.
> > To
> > > be
> > > > > > > honest
> > > > > > > > I believe they were already sorted out, but your e-mail makes
> > me
> > > > > think
> > > > > > > that
> > > > > > > > is not the case ;). I let Kris to clarify that with you. My
> > view
> > > is
> > > > > > that
> > > > > > > > data-index is a way to query recently closed and active
> > processes
> > > > > (the
> > > > > > > key
> > > > > > > > here is the definition of recently, which in my opinion
> should
> > be
> > > > > > > > configurable)
> > > > > > > > But, besides that discussion and being pragmatic, keeping
> > > finishing
> > > > > > > process
> > > > > > > > instances "for a while" in DataIndex was the only way for
> users
> > > to
> > > > > > query
> > > > > > > > the result of straight through processes. That's a function
> > that
> > > > > cannot
> > > > > > > be
> > > > > > > > removed right now
> > > > > > > >
> > > > > > > > On Mon, Feb 19, 2024 at 1:33 PM Alex Porcelli <
> > > porcelli@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > if data index was supposed to provide snapshot view of the
> > > > process
> > > > > > > > > instance… why do we keep it after the process instance is
> > > > finished?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Feb 19, 2024 at 7:12 AM Francisco Javier Tirado
> > Sarti <
> > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > >
> > > > > > > > > > Hi Martin.
> > > > > > > > > > After taking a deeper look at this, I realize that the
> > > > behaviour
> > > > > is
> > > > > > > the
> > > > > > > > > > expected one.
> > > > > > > > > > Runtimes DB does not track the completed process instance
> > > > (that's
> > > > > > > what
> > > > > > > > > the
> > > > > > > > > > JDBCProcessInstances warn is telling us), but DataIndex,
> as
> > > > > > expected,
> > > > > > > > is
> > > > > > > > > > tracking it in processes and nodes table. And yes it will
> > > grow
> > > > > over
> > > > > > > > time.
> > > > > > > > > > What we need is some configurable purge mechanism for
> > > > DataIndex,
> > > > > so
> > > > > > > it
> > > > > > > > > > eventually removes older completed process instances.
> > > > > > > > > >
> > > > > > > > > > On Tue, Feb 13, 2024 at 12:59 PM Francisco Javier Tirado
> > > Sarti
> > > > <
> > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Martin,
> > > > > > > > > > > Good catch!. Looks like the skipping performed for
> > process
> > > > > > > instances
> > > > > > > > is
> > > > > > > > > > > not applied to node instances. Something we definitely
> > need
> > > > to
> > > > > > > review
> > > > > > > > > on
> > > > > > > > > > > the runtimes side.
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Feb 12, 2024 at 11:59 PM Martin Weiler
> > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> On a somewhat related note, testing a simple workflow
> > > (start
> > > > > ->
> > > > > > > > script
> > > > > > > > > > >> node -> end), I see the following messages in the
> logs:
> > > > > > > > > > >> 2024-02-12 22:49:50,493 28758dde544c WARN
> > > > > > > > > > >>
> > [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> > > > > > > > > > >> (executor-thread-3) Skipping create of process
> instance
> > > id:
> > > > > > > > > > >> 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2
> > > > > > > > > > >>
> > > > > > > > > > >> So far, so good. And I'd expect to see no trace of
> this
> > > > > process
> > > > > > in
> > > > > > > > the
> > > > > > > > > > >> database if I don't have data audit enabled.
> > > > > > > > > > >>
> > > > > > > > > > >> However, the 'processes' table contains a row with
> > > state=2,
> > > > > with
> > > > > > > > > related
> > > > > > > > > > >> entries in the 'nodes' table. In a load test, I see
> > these
> > > > > tables
> > > > > > > > grow
> > > > > > > > > > >> significantly over time. Am I missing something to
> have
> > > > these
> > > > > > > > entries
> > > > > > > > > > >> cleaned up automatically?
> > > > > > > > > > >>
> > > > > > > > > > >> ________________________________________
> > > > > > > > > > >> From: Martin Weiler <mw...@ibm.com.INVALID>
> > > > > > > > > > >> Sent: Monday, February 12, 2024 3:40 PM
> > > > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > > > >> Subject: [EXTERNAL] RE: [DISCUSSION] Performance
> issues
> > > with
> > > > > > > > > data-index
> > > > > > > > > > >> persistence addon
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks everyone for your input. Based on this
> > discussion,
> > > I
> > > > > > opened
> > > > > > > > the
> > > > > > > > > > >> following PR:
> > > > > > > > > > >>
> > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1985
> > > > > > > > > > >>
> > > > > > > > > > >> With this change, the performance seems to be stable
> > over
> > > > > time:
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
> > > > > > > > > > >>
> > > > > > > > > > >> Martin
> > > > > > > > > > >>
> > > > > > > > > > >> ________________________________________
> > > > > > > > > > >> From: Gonzalo Muñoz <gm...@apache.org>
> > > > > > > > > > >> Sent: Friday, February 9, 2024 9:42 AM
> > > > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > > > >> Subject: [EXTERNAL] Re: [DISCUSSION] Performance
> issues
> > > with
> > > > > > > > > data-index
> > > > > > > > > > >> persistence addon
> > > > > > > > > > >>
> > > > > > > > > > >> Great work Francisco,
> > > > > > > > > > >> Martin, take a look at this link with some related
> tips
> > > (in
> > > > > case
> > > > > > > you
> > > > > > > > > > find
> > > > > > > > > > >> it useful):
> > > > > > > > > > >>
> > > > > https://www.cybertec-postgresql.com/en/index-your-foreign-key/
> > > > > > > > > > >>
> > > > > > > > > > >> El vie, 9 feb 2024 a las 17:20, Francisco Javier
> Tirado
> > > > Sarti
> > > > > (<
> > > > > > > > > > >> ftirados@redhat.com>) escribió:
> > > > > > > > > > >>
> > > > > > > > > > >> > For the moment being, we will keep JPA till we
> exhaust
> > > all
> > > > > > > > > > >> possibilities,
> > > > > > > > > > >> > let's call switching from jpa to jdbc our hidden
> plan
> > B
> > > ;)
> > > > > > > > > > >> > I already told Martin, but in order everyone to
> know,
> > > just
> > > > > > after
> > > > > > > > > > writing
> > > > > > > > > > >> > the previous email, I thought "what if Postgres is
> not
> > > > > > > > automatically
> > > > > > > > > > >> > indexing foreign keys like mysql?" and, eureka
> > > > > > > > > > >> > Postgres doc
> > > > > > > > > > >> >
> > > > > https://www.postgresql.org/docs/current/ddl-constraints.html
> > > > > > > > > > >> > Mysql doc
> > > > > > > > > > >> >
> > > > > > > >
> > > > https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> > > > > > > > > > >> > These are the relevant excerpt
> > > > > > > > > > >> >
> > > > > > > > > > >> > *Postgresql*
> > > > > > > > > > >> > *A foreign key must reference columns that either
> are
> > a
> > > > > > primary
> > > > > > > > key
> > > > > > > > > or
> > > > > > > > > > >> form
> > > > > > > > > > >> > a unique constraint, or are columns from a
> non-partial
> > > > > unique
> > > > > > > > index.
> > > > > > > > > > >> This
> > > > > > > > > > >> > means that the referenced columns always have an
> index
> > > to
> > > > > > allow
> > > > > > > > > > >> efficient
> > > > > > > > > > >> > lookups on whether a referencing row has a match.
> > Since
> > > a
> > > > > > DELETE
> > > > > > > > of
> > > > > > > > > a
> > > > > > > > > > >> row
> > > > > > > > > > >> > from the referenced table or an UPDATE of a
> referenced
> > > > > column
> > > > > > > will
> > > > > > > > > > >> require
> > > > > > > > > > >> > a scan of the referencing table for rows matching
> the
> > > old
> > > > > > value,
> > > > > > > > it
> > > > > > > > > is
> > > > > > > > > > >> > often a good idea to index the referencing columns
> > too.
> > > > > > Because
> > > > > > > > this
> > > > > > > > > > is
> > > > > > > > > > >> not
> > > > > > > > > > >> > always needed, and there are many choices available
> on
> > > how
> > > > > to
> > > > > > > > index,
> > > > > > > > > > the
> > > > > > > > > > >> > declaration of a foreign key constraint does not
> > > > > automatically
> > > > > > > > > create
> > > > > > > > > > an
> > > > > > > > > > >> > index on the referencing columns.*
> > > > > > > > > > >> > *Mysql*
> > > > > > > > > > >> > *MySQL requires that foreign key columns be indexed;
> > if
> > > > you
> > > > > > > > create a
> > > > > > > > > > >> table
> > > > > > > > > > >> > with a foreign key constraint but no index on a
> given
> > > > > column,
> > > > > > an
> > > > > > > > > index
> > > > > > > > > > >> is
> > > > > > > > > > >> > created. *
> > > > > > > > > > >> >
> > > > > > > > > > >> > So I asked Martin to especially create an index for
> > > > > > > > > > process_instance_id
> > > > > > > > > > >> > column on nodes table
> > > > > > > > > > >> > I think that will fix the problem detected on the
> > thread
> > > > > dump.
> > > > > > > > > > >> > The simpler process test to verify queries are fine
> > > still
> > > > > > > stands,
> > > > > > > > > > >> though ;)
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <
> > > > > > > tzimanyi@apache.org
> > > > > > > > >
> > > > > > > > > > >> wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> > > I always preferred pure JDBC over Hibernate
> myself,
> > > just
> > > > > for
> > > > > > > the
> > > > > > > > > > sake
> > > > > > > > > > >> of
> > > > > > > > > > >> > > control of what is happening :) So I would not -1
> > that
> > > > > > myself.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Tibor
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado
> > > Sarti <
> > > > > > > > > > >> > > ftirados@redhat.com>
> > > > > > > > > > >> > > napísal(a):
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > Hi,
> > > > > > > > > > >> > > > Usually I do not want to talk about work in
> > progress
> > > > > > because
> > > > > > > > > > >> > preliminary
> > > > > > > > > > >> > > > conclusions are pretty volatile but, well, there
> > > are a
> > > > > > > couple
> > > > > > > > of
> > > > > > > > > > >> things
> > > > > > > > > > >> > > > that can be concluded from the really valuable
> > > > > information
> > > > > > > > that
> > > > > > > > > > >> Martin
> > > > > > > > > > >> > > > provided.
> > > > > > > > > > >> > > > 1) In order to be able to determine if the
> number
> > of
> > > > > > > > statements
> > > > > > > > > is
> > > > > > > > > > >> > larger
> > > > > > > > > > >> > > > than expected, I asked Martin to test with a
> > simpler
> > > > > > process
> > > > > > > > > > >> > definition.
> > > > > > > > > > >> > > > One with just three nodes: start, script and
> end.
> > > The
> > > > > > script
> > > > > > > > one
> > > > > > > > > > >> should
> > > > > > > > > > >> > > > change just one variable. This way we can
> analyze
> > if
> > > > the
> > > > > > > > number
> > > > > > > > > of
> > > > > > > > > > >> > > queries
> > > > > > > > > > >> > > > is the expected one. From the single log (audit
> > was
> > > > > > > activated
> > > > > > > > > > them)
> > > > > > > > > > >> my
> > > > > > > > > > >> > > > conclusion is that the number of insert/updates
> > over
> > > > > > > processes
> > > > > > > > > and
> > > > > > > > > > >> > nodes
> > > > > > > > > > >> > > > (there a lot over task, that I will prefer to
> skip
> > > for
> > > > > > now,
> > > > > > > > baby
> > > > > > > > > > >> steps)
> > > > > > > > > > >> > > is
> > > > > > > > > > >> > > > the expected one.
> > > > > > > > > > >> > > > 2) Analysing the thread dump, we see around 15
> > > threads
> > > > > > > > executing
> > > > > > > > > > >> this
> > > > > > > > > > >> > > line
> > > > > > > > > > >> > > > at
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > > > > > > > > >> > > > so its pretty clear the code to be optimized ;).
> > I'm
> > > > > > > > evaluating
> > > > > > > > > > >> > > > possibilities within JPA/Hibernate, but I'm
> > starting
> > > > to
> > > > > > > think
> > > > > > > > > that
> > > > > > > > > > >> it
> > > > > > > > > > >> > > might
> > > > > > > > > > >> > > > be better to switch to JDBC and skip hibernate.
> > Our
> > > > > lives
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > > >> > > simpler,
> > > > > > > > > > >> > > > especially with a schema relatively simple like
> > ours
> > > > > (that
> > > > > > > > will
> > > > > > > > > be
> > > > > > > > > > >> my
> > > > > > > > > > >> > > > recommendation if I was an external consultant)
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <
> > > > > > > > > tzimanyi@apache.org
> > > > > > > > > > >
> > > > > > > > > > >> > > wrote:
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > > Hi,
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > this will be a bit off-topic. However as far
> as
> > > > > > > > performance, I
> > > > > > > > > > >> think
> > > > > > > > > > >> > we
> > > > > > > > > > >> > > > > should think about that we have string primary
> > > keys
> > > > > > > (IDs). I
> > > > > > > > > > would
> > > > > > > > > > >> > > expect
> > > > > > > > > > >> > > > > the database systems are much better with
> > indexing
> > > > > > numeric
> > > > > > > > > keys
> > > > > > > > > > >> than
> > > > > > > > > > >> > > > > strings. I remember from the past, when I was
> > > > working
> > > > > > with
> > > > > > > > > DBs,
> > > > > > > > > > >> that
> > > > > > > > > > >> > > > using
> > > > > > > > > > >> > > > > strings as keys or indexes was a discouraged
> > > > practice.
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > Best regards,
> > > > > > > > > > >> > > > > Tibor
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler
> > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > >
> > > > > > > > > > >> > > > > napísal(a):
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > > I changed the test to use MongoDB [1] and I
> > > don't
> > > > > see
> > > > > > a
> > > > > > > > > > >> performance
> > > > > > > > > > >> > > > > > degradation with this setup [2].
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > Please keep us posted of your findings.
> > Thanks!
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > Martin
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > [1]
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> >
> > > > > > > > > >
> > > > > > >
> > > >
> https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > > > > > > > >> > > > > > [2]
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > ________________________________________
> > > > > > > > > > >> > > > > > From: Francisco Javier Tirado Sarti <
> > > > > > > ftirados@redhat.com>
> > > > > > > > > > >> > > > > > Sent: Wednesday, February 7, 2024 11:40 AM
> > > > > > > > > > >> > > > > > To: dev@kie.apache.org
> > > > > > > > > > >> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION]
> > Performance
> > > > > > issues
> > > > > > > > with
> > > > > > > > > > >> > > data-index
> > > > > > > > > > >> > > > > > persistence addon
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > yes, it can be index degradation because of
> > > size,
> > > > > but
> > > > > > I
> > > > > > > > > > believe
> > > > > > > > > > >> (I
> > > > > > > > > > >> > > > might
> > > > > > > > > > >> > > > > be
> > > > > > > > > > >> > > > > > wrong) the db is too small (yet) for that.
> > > > > > > > > > >> > > > > > But, eventually, Postgres, when the DB is
> huge
> > > > > enough,
> > > > > > > > > > >> unavoidably
> > > > > > > > > > >> > > will
> > > > > > > > > > >> > > > > > behave like the graphic that Martin sent.
> > > > > > > > > > >> > > > > > Since I believe we are not huge enough
> (yet),
> > > lets
> > > > > > rule
> > > > > > > > out
> > > > > > > > > > >> another
> > > > > > > > > > >> > > > issue
> > > > > > > > > > >> > > > > > by analysing the sql logs (I requested those
> > to
> > > > > Martin
> > > > > > > > > offline
> > > > > > > > > > >> and
> > > > > > > > > > >> > he
> > > > > > > > > > >> > > > is
> > > > > > > > > > >> > > > > > going to kindly collect them).
> > > > > > > > > > >> > > > > > Also Im curious to know if Mongo behave in
> the
> > > > same
> > > > > > way.
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique
> > Gonzalez
> > > > > > > Martinez <
> > > > > > > > > > >> > > > > > egonzalez@apache.org> wrote:
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > > Hi Francisco,
> > > > > > > > > > >> > > > > > > I would highly recommend to check indexes
> > and
> > > > how
> > > > > > the
> > > > > > > > > > updates
> > > > > > > > > > >> > work
> > > > > > > > > > >> > > in
> > > > > > > > > > >> > > > > > data
> > > > > > > > > > >> > > > > > > index to avoid full scan table and lock
> the
> > > full
> > > > > > > table.
> > > > > > > > > Some
> > > > > > > > > > >> db
> > > > > > > > > > >> > are
> > > > > > > > > > >> > > > > very
> > > > > > > > > > >> > > > > > > sensitive to that.
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > El mié, 7 feb 2024, 18:41, Francisco
> Javier
> > > > Tirado
> > > > > > > > Sarti <
> > > > > > > > > > >> > > > > > > ftirados@redhat.com> escribió:
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > > Hi Martin,
> > > > > > > > > > >> > > > > > > > While I analyze the data, let me ask you
> > if
> > > it
> > > > > is
> > > > > > > > > possible
> > > > > > > > > > >> to
> > > > > > > > > > >> > > > perform
> > > > > > > > > > >> > > > > > > > another check (similar in a way to
> > disabling
> > > > > > > > data-index
> > > > > > > > > > like
> > > > > > > > > > >> > you
> > > > > > > > > > >> > > > do)
> > > > > > > > > > >> > > > > > Can
> > > > > > > > > > >> > > > > > > > you switch to MongoDB persistence and
> > check
> > > if
> > > > > the
> > > > > > > > same
> > > > > > > > > > >> > > degradation
> > > > > > > > > > >> > > > > > that
> > > > > > > > > > >> > > > > > > is
> > > > > > > > > > >> > > > > > > > there for postgres remains?
> > > > > > > > > > >> > > > > > > > I do not know if this is feasible but
> will
> > > > > > certainly
> > > > > > > > > > >> indicate
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > > > problem
> > > > > > > > > > >> > > > > > > > is on the postgres storage layer and I
> do
> > > not
> > > > > > have a
> > > > > > > > > clear
> > > > > > > > > > >> > > > prediction
> > > > > > > > > > >> > > > > > of
> > > > > > > > > > >> > > > > > > > what we will see when doing this switch.
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin
> > Weiler
> > > > > > > > > > >> > > > <mweiler@ibm.com.invalid
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > > > wrote:
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > > Hi Francisco,
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > thanks for your work on this important
> > > > topic!
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > I would like to share some test
> results
> > > > here,
> > > > > > > which
> > > > > > > > > > might
> > > > > > > > > > >> > help
> > > > > > > > > > >> > > to
> > > > > > > > > > >> > > > > > > improve
> > > > > > > > > > >> > > > > > > > > the codebase even further. I am using
> > the
> > > > > jmeter
> > > > > > > > based
> > > > > > > > > > >> test
> > > > > > > > > > >> > > case
> > > > > > > > > > >> > > > > from
> > > > > > > > > > >> > > > > > > > Pere
> > > > > > > > > > >> > > > > > > > > and Enrique (thanks guys!) [1] which
> > uses
> > > a
> > > > > load
> > > > > > > of
> > > > > > > > 30
> > > > > > > > > > >> > threads
> > > > > > > > > > >> > > to
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > 1) start a new process instance (POST)
> > > > > > > > > > >> > > > > > > > > 2) retrieve tasks for a user (GET)
> > > > > > > > > > >> > > > > > > > > 3) fetches task details (GET)
> > > > > > > > > > >> > > > > > > > > 4) complete a task (POST)
> > > > > > > > > > >> > > > > > > > > 5) execute a query on data-audit
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > With this test setup, I noticed that
> the
> > > > > > > performance
> > > > > > > > > for
> > > > > > > > > > >> the
> > > > > > > > > > >> > > POST
> > > > > > > > > > >> > > > > > > > > requests, in particular the one to
> > start a
> > > > new
> > > > > > > > process
> > > > > > > > > > >> > > instance,
> > > > > > > > > > >> > > > > > > degrades
> > > > > > > > > > >> > > > > > > > > over time - see graph [2]. If I run
> the
> > > same
> > > > > > test
> > > > > > > > > > without
> > > > > > > > > > >> > > > > data-index,
> > > > > > > > > > >> > > > > > > > then
> > > > > > > > > > >> > > > > > > > > there is no such performance
> degradation
> > > > [3].
> > > > > > You
> > > > > > > > can
> > > > > > > > > > >> find a
> > > > > > > > > > >> > > > thread
> > > > > > > > > > >> > > > > > > dump
> > > > > > > > > > >> > > > > > > > > captured a few minutes into the first
> > test
> > > > > here
> > > > > > > [4]
> > > > > > > > > that
> > > > > > > > > > >> > might
> > > > > > > > > > >> > > > help
> > > > > > > > > > >> > > > > > to
> > > > > > > > > > >> > > > > > > > see
> > > > > > > > > > >> > > > > > > > > some of the contention points.
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > I'd appreciate if you could take a
> look
> > > and
> > > > > see
> > > > > > if
> > > > > > > > > there
> > > > > > > > > > >> is
> > > > > > > > > > >> > > > > something
> > > > > > > > > > >> > > > > > > > that
> > > > > > > > > > >> > > > > > > > > can be further improved based on your
> > > > previous
> > > > > > > work.
> > > > > > > > > If
> > > > > > > > > > >> you
> > > > > > > > > > >> > > need
> > > > > > > > > > >> > > > > any
> > > > > > > > > > >> > > > > > > > > additional data, let me know, but
> > > otherwise
> > > > it
> > > > > > is
> > > > > > > > > > >> > > straightforward
> > > > > > > > > > >> > > > > to
> > > > > > > > > > >> > > > > > > run
> > > > > > > > > > >> > > > > > > > > the jmeter test as well.
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > Thanks,
> > > > > > > > > > >> > > > > > > > > Martin
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > [1]
> > > > > > > > > > >>
> https://github.com/pefernan/job-service-refactor-test/
> > > > > > > > > > >> > > > > > > > > [2]
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > > > > > > >> > > > > > > > > [3]
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > > > > > > >> > > > > > > > > [4]
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > >
> ________________________________________
> > > > > > > > > > >> > > > > > > > > From: Francisco Javier Tirado Sarti <
> > > > > > > > > > ftirados@redhat.com>
> > > > > > > > > > >> > > > > > > > > Sent: Wednesday, January 17, 2024 9:13
> > AM
> > > > > > > > > > >> > > > > > > > > To: dev@kie.apache.org
> > > > > > > > > > >> > > > > > > > > Cc: Pere Fernandez Perez
> > > > > > > > > > >> > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION]
> > > > > Performance
> > > > > > > > > issues
> > > > > > > > > > >> with
> > > > > > > > > > >> > > > > > data-index
> > > > > > > > > > >> > > > > > > > > persistence addon
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > Hi Alex,
> > > > > > > > > > >> > > > > > > > > I did not take times (which depends
> on a
> > > > > number
> > > > > > of
> > > > > > > > > > >> variables
> > > > > > > > > > >> > > that
> > > > > > > > > > >> > > > > > > > > drastically change between
> > environments),
> > > > but
> > > > > > > verify
> > > > > > > > > > that
> > > > > > > > > > >> the
> > > > > > > > > > >> > > > > number
> > > > > > > > > > >> > > > > > of
> > > > > > > > > > >> > > > > > > > > updates has been reduced drastically
> > > without
> > > > > > > losing
> > > > > > > > > > >> > > > functionality,
> > > > > > > > > > >> > > > > > > which
> > > > > > > > > > >> > > > > > > > is
> > > > > > > > > > >> > > > > > > > > objectively a good thing. If before
> the
> > > > > change,
> > > > > > > for
> > > > > > > > > > every
> > > > > > > > > > >> > node
> > > > > > > > > > >> > > > > > > executed,
> > > > > > > > > > >> > > > > > > > we
> > > > > > > > > > >> > > > > > > > > have an update for every node
> previously
> > > > > > executed,
> > > > > > > > so
> > > > > > > > > > if a
> > > > > > > > > > >> > > > process
> > > > > > > > > > >> > > > > > have
> > > > > > > > > > >> > > > > > > > 50
> > > > > > > > > > >> > > > > > > > > nodes to execute, we were performing
> > > nearly
> > > > > > > 50*51/2
> > > > > > > > > > >> updates,
> > > > > > > > > > >> > > > which
> > > > > > > > > > >> > > > > > > gives
> > > > > > > > > > >> > > > > > > > us
> > > > > > > > > > >> > > > > > > > > a total of  1275 updates, now we have
> > just
> > > > one
> > > > > > for
> > > > > > > > > every
> > > > > > > > > > >> node
> > > > > > > > > > >> > > > being
> > > > > > > > > > >> > > > > > > > > executed, implying a total of 50
> > updates.
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex
> > > > Porcelli
> > > > > <
> > > > > > > > > > >> > > alex@porcelli.me>
> > > > > > > > > > >> > > > > > > wrote:
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > Francisco,
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > I noticed that your PR has been
> > merged,
> > > > but
> > > > > I
> > > > > > > was
> > > > > > > > > > >> expecting
> > > > > > > > > > >> > > (at
> > > > > > > > > > >> > > > > > least
> > > > > > > > > > >> > > > > > > > > > was my understanding from this
> thread)
> > > > that
> > > > > > > before
> > > > > > > > > > >> merging
> > > > > > > > > > >> > > some
> > > > > > > > > > >> > > > > > > > > > benchmark data would be shared in
> > > advance
> > > > -
> > > > > to
> > > > > > > > > assess
> > > > > > > > > > >> the
> > > > > > > > > > >> > > > > > > cost/benefit
> > > > > > > > > > >> > > > > > > > > > of such a decent size change.
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > Do you have any information to
> share?
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM
> > > Francisco
> > > > > > Javier
> > > > > > > > > > Tirado
> > > > > > > > > > >> > Sarti
> > > > > > > > > > >> > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > Yes, as intended, now we have one
> > > select
> > > > > and
> > > > > > > one
> > > > > > > > > > >> > > > insert/update
> > > > > > > > > > >> > > > > > per
> > > > > > > > > > >> > > > > > > > node
> > > > > > > > > > >> > > > > > > > > > > event.
> > > > > > > > > > >> > > > > > > > > > > I moved the PR as ready for review
> > and
> > > > > give
> > > > > > > > @Pere
> > > > > > > > > > >> > Fernandez
> > > > > > > > > > >> > > > > Perez
> > > > > > > > > > >> > > > > > > > > > > <pe...@redhat.com> permission
> to
> > > the
> > > > > > > branch
> > > > > > > > so
> > > > > > > > > > he
> > > > > > > > > > >> can
> > > > > > > > > > >> > > > edit
> > > > > > > > > > >> > > > > it
> > > > > > > > > > >> > > > > > > in
> > > > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >> > > > > > > > > > > next two weeks (Ill be on PTO)  if
> > > > > desired,
> > > > > > > > before
> > > > > > > > > > >> > merging.
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM
> Alex
> > > > > > Porcelli
> > > > > > > <
> > > > > > > > > > >> > > > > alex@porcelli.me>
> > > > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > Cool, thank you Francisco!
> > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > Did you manage to get some
> > > preliminary
> > > > > > data
> > > > > > > > > about
> > > > > > > > > > >> > > > > improvements?
> > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM
> > > > > Francisco
> > > > > > > > > Javier
> > > > > > > > > > >> > Tirado
> > > > > > > > > > >> > > > > Sarti
> > > > > > > > > > >> > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > Yes, after some delay because
> of
> > > > > > quarkus 3
> > > > > > > > > > >> migration.
> > > > > > > > > > >> > > Im
> > > > > > > > > > >> > > > > > > refining
> > > > > > > > > > >> > > > > > > > > > this
> > > > > > > > > > >> > > > > > > > > > > > > draft PR
> > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > >
> > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > On Thu, Dec 21, 2023 at
> 5:48 PM
> > > Alex
> > > > > > > > Porcelli
> > > > > > > > > <
> > > > > > > > > > >> > > > > > > alex@porcelli.me>
> > > > > > > > > > >> > > > > > > > > > wrote:
> > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > Any update or new findings
> on
> > > this
> > > > > > > topic?
> > > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > On Tue, Nov 28, 2023 at
> > 8:38 AM
> > > > > > > Francisco
> > > > > > > > > > Javier
> > > > > > > > > > >> > > Tirado
> > > > > > > > > > >> > > > > > Sarti
> > > > > > > > > > >> > > > > > > > > > > > > > <ft...@redhat.com>
> wrote:
> > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > > Hi Alex,
> > > > > > > > > > >> > > > > > > > > > > > > > > After considering
> different
> > > > > options
> > > > > > to
> > > > > > > > > > improve
> > > > > > > > > > >> > > > > > performance,
> > > > > > > > > > >> > > > > > > > we
> > > > > > > > > > >> > > > > > > > > > feel
> > > > > > > > > > >> > > > > > > > > > > > that
> > > > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > > > > >> > > > > > > > > > > > > > > is time to "partially"
> move
> > > away
> > > > > > from
> > > > > > > > the
> > > > > > > > > > >> current
> > > > > > > > > > >> > > Map
> > > > > > > > > > >> > > > > > style
> > > > > > > > > > >> > > > > > > > > > > > interface (
> > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > > > > > >> > > > > > > > > > > > > > )
> > > > > > > > > > >> > > > > > > > > > > > > > > which was shared with
> > Trusty,
> > > to
> > > > > one
> > > > > > > > more
> > > > > > > > > > >> > suitable
> > > > > > > > > > >> > > > for
> > > > > > > > > > >> > > > > > > usage
> > > > > > > > > > >> > > > > > > > > > with a
> > > > > > > > > > >> > > > > > > > > > > > > > > relational DB like
> > postgresql
> > > > (but
> > > > > > > still
> > > > > > > > > > >> > compatible
> > > > > > > > > > >> > > > > with
> > > > > > > > > > >> > > > > > > big
> > > > > > > > > > >> > > > > > > > > > table
> > > > > > > > > > >> > > > > > > > > > > > dbs).
> > > > > > > > > > >> > > > > > > > > > > > > > > The idea will be to
> replace
> > > > > generic
> > > > > > > > > Storage
> > > > > > > > > > >> > > interface
> > > > > > > > > > >> > > > > by
> > > > > > > > > > >> > > > > > > four
> > > > > > > > > > >> > > > > > > > > > > > specific
> > > > > > > > > > >> > > > > > > > > > > > > > > interfaces (which will
> > inherit
> > > > > from
> > > > > > a
> > > > > > > > > common
> > > > > > > > > > >> one
> > > > > > > > > > >> > > that
> > > > > > > > > > >> > > > > > keeps
> > > > > > > > > > >> > > > > > > > the
> > > > > > > > > > >> > > > > > > > > > query
> > > > > > > > > > >> > > > > > > > > > > > > > part
> > > > > > > > > > >> > > > > > > > > > > > > > > at is it. with get and
> query
> > > > > > methods),
> > > > > > > > > that
> > > > > > > > > > >> will
> > > > > > > > > > >> > > > > include
> > > > > > > > > > >> > > > > > > the
> > > > > > > > > > >> > > > > > > > > > required
> > > > > > > > > > >> > > > > > > > > > > > > > > modification operations
> for
> > > the
> > > > > four
> > > > > > > > > > DataIndex
> > > > > > > > > > >> > > > > "domains":
> > > > > > > > > > >> > > > > > > > > > > > > > processinstance,
> > > > > > > > > > >> > > > > > > > > > > > > > > usertask,
> processdefinitions
> > > and
> > > > > > jobs.
> > > > > > > > > Those
> > > > > > > > > > >> > > > interfaces
> > > > > > > > > > >> > > > > > > will
> > > > > > > > > > >> > > > > > > > > > define
> > > > > > > > > > >> > > > > > > > > > > > > > methods
> > > > > > > > > > >> > > > > > > > > > > > > > > like addNode, addVariable,
> > > > > > updateTask,
> > > > > > > > > > >> > > > > addAttachment.....
> > > > > > > > > > >> > > > > > > > that
> > > > > > > > > > >> > > > > > > > > > will
> > > > > > > > > > >> > > > > > > > > > > > allow
> > > > > > > > > > >> > > > > > > > > > > > > > > the persistent layer
> > > > > implementation
> > > > > > > to
> > > > > > > > > just
> > > > > > > > > > >> > update
> > > > > > > > > > >> > > > the
> > > > > > > > > > >> > > > > > > > needed
> > > > > > > > > > >> > > > > > > > > > info
> > > > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > > >> > > > > > > > > > > > > > > DB  (for example, for
> > addNode
> > > in
> > > > > > > > Postgres,
> > > > > > > > > > >> just
> > > > > > > > > > >> > > > insert
> > > > > > > > > > >> > > > > a
> > > > > > > > > > >> > > > > > > row
> > > > > > > > > > >> > > > > > > > > into
> > > > > > > > > > >> > > > > > > > > > > > nodes
> > > > > > > > > > >> > > > > > > > > > > > > > > table, for addNode in
> Mongo,
> > > > > > basically
> > > > > > > > the
> > > > > > > > > > >> same
> > > > > > > > > > >> > > > atomic
> > > > > > > > > > >> > > > > > > upsert
> > > > > > > > > > >> > > > > > > > > > > > operation
> > > > > > > > > > >> > > > > > > > > > > > > > > that is currently done).
> > > > > Therefore,
> > > > > > we
> > > > > > > > > > >> increase
> > > > > > > > > > >> > > > > > performance
> > > > > > > > > > >> > > > > > > > for
> > > > > > > > > > >> > > > > > > > > > > > Postgres
> > > > > > > > > > >> > > > > > > > > > > > > > > and keep the current one
> for
> > > > > Mongo.
> > > > > > > The
> > > > > > > > > > >> current
> > > > > > > > > > >> > DB
> > > > > > > > > > >> > > > > > schemas
> > > > > > > > > > >> > > > > > > > > won't
> > > > > > > > > > >> > > > > > > > > > be
> > > > > > > > > > >> > > > > > > > > > > > > > > touched.
> > > > > > > > > > >> > > > > > > > > > > > > > > Since the code change is
> > > large,
> > > > I
> > > > > do
> > > > > > > not
> > > > > > > > > > think
> > > > > > > > > > >> > I'll
> > > > > > > > > > >> > > > be
> > > > > > > > > > >> > > > > > able
> > > > > > > > > > >> > > > > > > > to
> > > > > > > > > > >> > > > > > > > > > have
> > > > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > > >> > > > > > > > > > > > > > PR
> > > > > > > > > > >> > > > > > > > > > > > > > > ready till next week.
> > > > > > > > > > >> > > > > > > > > > > > > > > But before starting,
> please
> > > let
> > > > me
> > > > > > > know
> > > > > > > > if
> > > > > > > > > > >> that
> > > > > > > > > > >> > > > > approach
> > > > > > > > > > >> > > > > > is
> > > > > > > > > > >> > > > > > > > > fine
> > > > > > > > > > >> > > > > > > > > > for
> > > > > > > > > > >> > > > > > > > > > > > you.
> > > > > > > > > > >> > > > > > > > > > > > > > > Best regards.
> > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> > > 6:55 PM
> > > > > Alex
> > > > > > > > > > Porcelli
> > > > > > > > > > >> <
> > > > > > > > > > >> > > > > > > > > alex@porcelli.me>
> > > > > > > > > > >> > > > > > > > > > > > wrote:
> > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > > > Thank you Francisco to
> > > getting
> > > > > > > deeper
> > > > > > > > on
> > > > > > > > > > >> this…
> > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > > > Looking forward to see
> the
> > > > > results
> > > > > > > of
> > > > > > > > > your
> > > > > > > > > > >> > > > suggested
> > > > > > > > > > >> > > > > > > > > > improvements.
> > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> > > > 9:40 AM
> > > > > > > > > Francisco
> > > > > > > > > > >> > Javier
> > > > > > > > > > >> > > > > Tirado
> > > > > > > > > > >> > > > > > > > > Sarti <
> > > > > > > > > > >> > > > > > > > > > > > > > > > ftirados@redhat.com>
> > wrote:
> > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > > > > I forgot to attach the
> > > > queries
> > > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > > > > On Fri, Nov 24, 2023
> at
> > > > > 3:04 PM
> > > > > > > > > > Francisco
> > > > > > > > > > >> > > Javier
> > > > > > > > > > >> > > > > > Tirado
> > > > > > > > > > >> > > > > > > > > > Sarti <
> > > > > > > > > > >> > > > > > > > > > > > > > > > > ftirados@redhat.com>
> > > wrote:
> > > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > > > >> Hi,
> > > > > > > > > > >> > > > > > > > > > > > > > > > >> A brief update on
> this
> > > > topic.
> > > > > > > > > > >> > > > > > > > > > > > > > > > >> After doing a simple
> > test
> > > > > with
> > > > > > > > > example
> > > > > > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > > > > >> > > > > > > > > > > > > > > > ,
> > > > > > > > > > >> > > > > > > > > > > > > > > > >> the number of updates
> > > over
> > > > > > Nodes
> > > > > > > > > table
> > > > > > > > > > is
> > > > > > > > > > >> > n*n,
> > > > > > > > > > >> > > > so
> > > > > > > > > > >> > > > > we
> > > > > > > > > > >> > > > > > > > > manage
> > > > > > > > > > >> > > > > > > > > > to
> > > > > > > > > > >> > > > > > > > > > > > > > obtain a
> > > > > > > > > > >> > > > > > > > > > > > > > > > >> perfect quadratic
> > > > performance
> > > > > > > > > > >> degradation.
> > > > > > > > > > >> > The
> > > > > > > > > > >> > > > > > problem
> > > > > > > > > > >> > > > > > > > is
> > > > > > > > > > >> > > > > > > > > > worse
> > > > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > > >> > > > > > > > > > > > > > > > case
> > > >

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Francisco Javier Tirado Sarti <ft...@redhat.com>]

Hi Enrique,
If we configure such policies through properties, it won't be enough to
define a naming convention and rely on the target platform configuration
capabilities (SpringBoot or Quarkus)?.

On Wed, Feb 21, 2024 at 12:36 PM Enrique Gonzalez Martinez <
egonzalez@apache.org> wrote:

> Hi Francisco
>
> The discussion we need to have before how to achieve certain features, is
> the overall of the user experience. If you want to clean it up that way it
> is fine by me. I have nothing against setting policy related to some sort
> of clean up for a process completed some time ago.
>
> As we discussed previously the data index is a snapshot of the last state
> of the process instance included completed, but nothing was said once
> completed when we need to clean that up so any policy is welcome. How to
> achieve that policy is something completely different.
>
> The main problem is how every microservice is exposing that API to the end
> user or consumer (other system) and which system is going to be the façade
> for complex deployments and operations. That is the discussion I was
> mentioning before.
>
> What I want to avoid is to set different policies among components and we
> should strive to be as much as possible to offer certain capabilities in
> the same fashion, e.g: clean up mechanism.
> In the same way I want to avoid making the current system more complex than
> they are. So far they are aligned offering one simple responsibility but I
> would be against mixing for instance job service with data index.
>
> El mié, 21 feb 2024 a las 12:13, Francisco Javier Tirado Sarti (<
> ftirados@redhat.com>) escribió:
>
> > Hi Enrique,
> > In the case of data index I think the data to be purged is finished
> process
> > instances (I do not think we should remove process instance that has been
> > alive for ages, even if it is very likely they are not going to be ever
> > completed)
> > Once you delete those process instances, you also delete the associated
> > user tasks and jobs.
> > Therefore the problem is relatively simple, to be able to configure how
> > much time a completed process instance should remain in the data index
> > database. We can take a simple approach: a property with a min duration
> > that cannot be changed once data index is started; a slightly complex
> one:
> > the same property but watching it to react for changes; or the full
> suite:
> > an admin API to be able to change the policy at any moment.
> > I think this discussion is worth having.
> >
> > On Wed, Feb 21, 2024 at 6:15 AM Enrique Gonzalez Martinez <
> > egonzalez@apache.org> wrote:
> >
> > > Hi Martin, the main problem regarding the purge is because it is still
> > > unclear the policy for what tech to use and the future components.
> > >
> > > Recently we had a discussion about proposing graphql for this sort of
> > admin
> > > tasks. So far for subsystems we have been using rest endpoints (like
> > update
> > > timers or modify human task or change processes). There is one
> exception
> > > which is the gateway that is pure graphql and somehow uses graphql for
> > > everything making complex operations under the hood.
> > >
> > > This has somehow frozen the purge for data audit for a bit and the
> > proposal
> > > was to use rest endpoints to do the clean up in the component and offer
> > the
> > > graphql counterpart in the gateway promoting it to a first class
> citizen
> > > component instead of having it embedded in the data index.
> > >
> > > I would suggest to come up at least with a policy first regarding the
> > > convention every component should address this.
> > >
> > >
> > > El mar, 20 feb 2024, 23:39, Martin Weiler <mw...@ibm.com.invalid>
> > > escribió:
> > >
> > > > IMO, it is good to have this discussion around data sanity now
> instead
> > of
> > > > putting it off until later when data has already accumulated in
> > > production
> > > > environments.
> > > >
> > > > Based on the input here, we are dealing with three types of data:
> > > > 1. Runtime data - active instances only, engine cleans up the data
> > > > automatically at process instance end
> > > > 2. Historic log data - data created by data-audit intended for long
> > term
> > > > storage
> > > > 3. Data-index data - somehow this data falls in between the two
> > > > aforementioned categories, with the idea of the data being "recent",
> > but
> > > > not restricted to active instances only
> > > >
> > > > We'd need purge strategies for both #2 and #3 (perhaps different
> ones,
> > or
> > > > with different config settings) in order to prevent unlimited data
> > > growth.
> > > >
> > > > ________________________________________
> > > > From: Enrique Gonzalez Martinez <eg...@apache.org>
> > > > Sent: Monday, February 19, 2024 7:11 AM
> > > > To: dev@kie.apache.org
> > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> data-index
> > > > persistence addon
> > > >
> > > > Hi Francisco,
> > > > To give you more context about this.
> > > >
> > > > STP is a concept, a process with certain constraints: no persistence
> > and
> > > > returning the outcome in the call (sync execution with no idle
> states).
> > > It
> > > > was a requirement from a user in the past. One of the requirements
> was
> > > > leaving no trail. In v7 was easy because you could disable the audit
> in
> > > > that case. Actually we have the same way to do what we did in v7 in
> > here
> > > as
> > > > you can add/remove index just removing deps.
> > > >
> > > > We have the same outcome with different approaches and STP is already
> > > > delivered.
> > > >
> > > > El lun, 19 feb 2024 a las 14:46, Francisco Javier Tirado Sarti (<
> > > > ftirados@redhat.com>) escribió:
> > > >
> > > > > Regarding STP (which is not a concept that we have in the code. I
> > mean
> > > > STP
> > > > > are processes as nonSTP are), I guess, as all processes, they were
> > kept
> > > > in
> > > > > DataIndex once completed because users wanted (and still wants) to
> > > check
> > > > > the result once the call had been performed. If we want to leave no
> > > trace
> > > > > of them in DataIndex for some reason, we will need to make it a
> > > > > Runtimes concept so DataIndex can handle them in a different way.
> > > > >
> > > > > On Mon, Feb 19, 2024 at 2:27 PM Enrique Gonzalez Martinez <
> > > > > egonzalez@apache.org> wrote:
> > > > >
> > > > > > Alex:
> > > > > > Right now the data index is working in the same way as it did in
> v7
> > > > with
> > > > > > the emitters. The only difference between two impl is that in
> here
> > > the
> > > > > > storage is pgsql instead elastic search.  You are right regarding
> > is
> > > a
> > > > > > snapshot of the last state of the process but we did never define
> > how
> > > > > long
> > > > > > would be alive that dats Honestly i am happy right now with the
> way
> > > it
> > > > > > works. The clean up mechanism is still tbd because we still need
> to
> > > > > discuss
> > > > > > other stuff first.
> > > > > >
> > > > > >
> > > > > > Regarding stp is to leave no trail because u can get the outcome
> > > > directly
> > > > > > from the call. It was defined like that in v7. So there is no use
> > for
> > > > the
> > > > > > index or the audit.
> > > > > >
> > > > > > El lun, 19 feb 2024, 14:13, Francisco Javier Tirado Sarti <
> > > > > > ftirados@redhat.com> escribió:
> > > > > >
> > > > > > > Hi Alex,
> > > > > > > There has been some confusion about the purpose of DataIndex.
> To
> > be
> > > > > > honest
> > > > > > > I believe they were already sorted out, but your e-mail makes
> me
> > > > think
> > > > > > that
> > > > > > > is not the case ;). I let Kris to clarify that with you. My
> view
> > is
> > > > > that
> > > > > > > data-index is a way to query recently closed and active
> processes
> > > > (the
> > > > > > key
> > > > > > > here is the definition of recently, which in my opinion should
> be
> > > > > > > configurable)
> > > > > > > But, besides that discussion and being pragmatic, keeping
> > finishing
> > > > > > process
> > > > > > > instances "for a while" in DataIndex was the only way for users
> > to
> > > > > query
> > > > > > > the result of straight through processes. That's a function
> that
> > > > cannot
> > > > > > be
> > > > > > > removed right now
> > > > > > >
> > > > > > > On Mon, Feb 19, 2024 at 1:33 PM Alex Porcelli <
> > porcelli@apache.org
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > if data index was supposed to provide snapshot view of the
> > > process
> > > > > > > > instance… why do we keep it after the process instance is
> > > finished?
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Feb 19, 2024 at 7:12 AM Francisco Javier Tirado
> Sarti <
> > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > >
> > > > > > > > > Hi Martin.
> > > > > > > > > After taking a deeper look at this, I realize that the
> > > behaviour
> > > > is
> > > > > > the
> > > > > > > > > expected one.
> > > > > > > > > Runtimes DB does not track the completed process instance
> > > (that's
> > > > > > what
> > > > > > > > the
> > > > > > > > > JDBCProcessInstances warn is telling us), but DataIndex, as
> > > > > expected,
> > > > > > > is
> > > > > > > > > tracking it in processes and nodes table. And yes it will
> > grow
> > > > over
> > > > > > > time.
> > > > > > > > > What we need is some configurable purge mechanism for
> > > DataIndex,
> > > > so
> > > > > > it
> > > > > > > > > eventually removes older completed process instances.
> > > > > > > > >
> > > > > > > > > On Tue, Feb 13, 2024 at 12:59 PM Francisco Javier Tirado
> > Sarti
> > > <
> > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > >
> > > > > > > > > > Hi Martin,
> > > > > > > > > > Good catch!. Looks like the skipping performed for
> process
> > > > > > instances
> > > > > > > is
> > > > > > > > > > not applied to node instances. Something we definitely
> need
> > > to
> > > > > > review
> > > > > > > > on
> > > > > > > > > > the runtimes side.
> > > > > > > > > >
> > > > > > > > > > On Mon, Feb 12, 2024 at 11:59 PM Martin Weiler
> > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> On a somewhat related note, testing a simple workflow
> > (start
> > > > ->
> > > > > > > script
> > > > > > > > > >> node -> end), I see the following messages in the logs:
> > > > > > > > > >> 2024-02-12 22:49:50,493 28758dde544c WARN
> > > > > > > > > >>
> [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> > > > > > > > > >> (executor-thread-3) Skipping create of process instance
> > id:
> > > > > > > > > >> 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2
> > > > > > > > > >>
> > > > > > > > > >> So far, so good. And I'd expect to see no trace of this
> > > > process
> > > > > in
> > > > > > > the
> > > > > > > > > >> database if I don't have data audit enabled.
> > > > > > > > > >>
> > > > > > > > > >> However, the 'processes' table contains a row with
> > state=2,
> > > > with
> > > > > > > > related
> > > > > > > > > >> entries in the 'nodes' table. In a load test, I see
> these
> > > > tables
> > > > > > > grow
> > > > > > > > > >> significantly over time. Am I missing something to have
> > > these
> > > > > > > entries
> > > > > > > > > >> cleaned up automatically?
> > > > > > > > > >>
> > > > > > > > > >> ________________________________________
> > > > > > > > > >> From: Martin Weiler <mw...@ibm.com.INVALID>
> > > > > > > > > >> Sent: Monday, February 12, 2024 3:40 PM
> > > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > > >> Subject: [EXTERNAL] RE: [DISCUSSION] Performance issues
> > with
> > > > > > > > data-index
> > > > > > > > > >> persistence addon
> > > > > > > > > >>
> > > > > > > > > >> Thanks everyone for your input. Based on this
> discussion,
> > I
> > > > > opened
> > > > > > > the
> > > > > > > > > >> following PR:
> > > > > > > > > >>
> > > https://github.com/apache/incubator-kie-kogito-apps/pull/1985
> > > > > > > > > >>
> > > > > > > > > >> With this change, the performance seems to be stable
> over
> > > > time:
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
> > > > > > > > > >>
> > > > > > > > > >> Martin
> > > > > > > > > >>
> > > > > > > > > >> ________________________________________
> > > > > > > > > >> From: Gonzalo Muñoz <gm...@apache.org>
> > > > > > > > > >> Sent: Friday, February 9, 2024 9:42 AM
> > > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > > >> Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues
> > with
> > > > > > > > data-index
> > > > > > > > > >> persistence addon
> > > > > > > > > >>
> > > > > > > > > >> Great work Francisco,
> > > > > > > > > >> Martin, take a look at this link with some related tips
> > (in
> > > > case
> > > > > > you
> > > > > > > > > find
> > > > > > > > > >> it useful):
> > > > > > > > > >>
> > > > https://www.cybertec-postgresql.com/en/index-your-foreign-key/
> > > > > > > > > >>
> > > > > > > > > >> El vie, 9 feb 2024 a las 17:20, Francisco Javier Tirado
> > > Sarti
> > > > (<
> > > > > > > > > >> ftirados@redhat.com>) escribió:
> > > > > > > > > >>
> > > > > > > > > >> > For the moment being, we will keep JPA till we exhaust
> > all
> > > > > > > > > >> possibilities,
> > > > > > > > > >> > let's call switching from jpa to jdbc our hidden plan
> B
> > ;)
> > > > > > > > > >> > I already told Martin, but in order everyone to know,
> > just
> > > > > after
> > > > > > > > > writing
> > > > > > > > > >> > the previous email, I thought "what if Postgres is not
> > > > > > > automatically
> > > > > > > > > >> > indexing foreign keys like mysql?" and, eureka
> > > > > > > > > >> > Postgres doc
> > > > > > > > > >> >
> > > > https://www.postgresql.org/docs/current/ddl-constraints.html
> > > > > > > > > >> > Mysql doc
> > > > > > > > > >> >
> > > > > > >
> > > https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> > > > > > > > > >> > These are the relevant excerpt
> > > > > > > > > >> >
> > > > > > > > > >> > *Postgresql*
> > > > > > > > > >> > *A foreign key must reference columns that either are
> a
> > > > > primary
> > > > > > > key
> > > > > > > > or
> > > > > > > > > >> form
> > > > > > > > > >> > a unique constraint, or are columns from a non-partial
> > > > unique
> > > > > > > index.
> > > > > > > > > >> This
> > > > > > > > > >> > means that the referenced columns always have an index
> > to
> > > > > allow
> > > > > > > > > >> efficient
> > > > > > > > > >> > lookups on whether a referencing row has a match.
> Since
> > a
> > > > > DELETE
> > > > > > > of
> > > > > > > > a
> > > > > > > > > >> row
> > > > > > > > > >> > from the referenced table or an UPDATE of a referenced
> > > > column
> > > > > > will
> > > > > > > > > >> require
> > > > > > > > > >> > a scan of the referencing table for rows matching the
> > old
> > > > > value,
> > > > > > > it
> > > > > > > > is
> > > > > > > > > >> > often a good idea to index the referencing columns
> too.
> > > > > Because
> > > > > > > this
> > > > > > > > > is
> > > > > > > > > >> not
> > > > > > > > > >> > always needed, and there are many choices available on
> > how
> > > > to
> > > > > > > index,
> > > > > > > > > the
> > > > > > > > > >> > declaration of a foreign key constraint does not
> > > > automatically
> > > > > > > > create
> > > > > > > > > an
> > > > > > > > > >> > index on the referencing columns.*
> > > > > > > > > >> > *Mysql*
> > > > > > > > > >> > *MySQL requires that foreign key columns be indexed;
> if
> > > you
> > > > > > > create a
> > > > > > > > > >> table
> > > > > > > > > >> > with a foreign key constraint but no index on a given
> > > > column,
> > > > > an
> > > > > > > > index
> > > > > > > > > >> is
> > > > > > > > > >> > created. *
> > > > > > > > > >> >
> > > > > > > > > >> > So I asked Martin to especially create an index for
> > > > > > > > > process_instance_id
> > > > > > > > > >> > column on nodes table
> > > > > > > > > >> > I think that will fix the problem detected on the
> thread
> > > > dump.
> > > > > > > > > >> > The simpler process test to verify queries are fine
> > still
> > > > > > stands,
> > > > > > > > > >> though ;)
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <
> > > > > > tzimanyi@apache.org
> > > > > > > >
> > > > > > > > > >> wrote:
> > > > > > > > > >> >
> > > > > > > > > >> > > I always preferred pure JDBC over Hibernate myself,
> > just
> > > > for
> > > > > > the
> > > > > > > > > sake
> > > > > > > > > >> of
> > > > > > > > > >> > > control of what is happening :) So I would not -1
> that
> > > > > myself.
> > > > > > > > > >> > >
> > > > > > > > > >> > > Tibor
> > > > > > > > > >> > >
> > > > > > > > > >> > > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado
> > Sarti <
> > > > > > > > > >> > > ftirados@redhat.com>
> > > > > > > > > >> > > napísal(a):
> > > > > > > > > >> > >
> > > > > > > > > >> > > > Hi,
> > > > > > > > > >> > > > Usually I do not want to talk about work in
> progress
> > > > > because
> > > > > > > > > >> > preliminary
> > > > > > > > > >> > > > conclusions are pretty volatile but, well, there
> > are a
> > > > > > couple
> > > > > > > of
> > > > > > > > > >> things
> > > > > > > > > >> > > > that can be concluded from the really valuable
> > > > information
> > > > > > > that
> > > > > > > > > >> Martin
> > > > > > > > > >> > > > provided.
> > > > > > > > > >> > > > 1) In order to be able to determine if the number
> of
> > > > > > > statements
> > > > > > > > is
> > > > > > > > > >> > larger
> > > > > > > > > >> > > > than expected, I asked Martin to test with a
> simpler
> > > > > process
> > > > > > > > > >> > definition.
> > > > > > > > > >> > > > One with just three nodes: start, script and end.
> > The
> > > > > script
> > > > > > > one
> > > > > > > > > >> should
> > > > > > > > > >> > > > change just one variable. This way we can analyze
> if
> > > the
> > > > > > > number
> > > > > > > > of
> > > > > > > > > >> > > queries
> > > > > > > > > >> > > > is the expected one. From the single log (audit
> was
> > > > > > activated
> > > > > > > > > them)
> > > > > > > > > >> my
> > > > > > > > > >> > > > conclusion is that the number of insert/updates
> over
> > > > > > processes
> > > > > > > > and
> > > > > > > > > >> > nodes
> > > > > > > > > >> > > > (there a lot over task, that I will prefer to skip
> > for
> > > > > now,
> > > > > > > baby
> > > > > > > > > >> steps)
> > > > > > > > > >> > > is
> > > > > > > > > >> > > > the expected one.
> > > > > > > > > >> > > > 2) Analysing the thread dump, we see around 15
> > threads
> > > > > > > executing
> > > > > > > > > >> this
> > > > > > > > > >> > > line
> > > > > > > > > >> > > > at
> > > > > > > > > >> > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > > > > > > > >> > > > so its pretty clear the code to be optimized ;).
> I'm
> > > > > > > evaluating
> > > > > > > > > >> > > > possibilities within JPA/Hibernate, but I'm
> starting
> > > to
> > > > > > think
> > > > > > > > that
> > > > > > > > > >> it
> > > > > > > > > >> > > might
> > > > > > > > > >> > > > be better to switch to JDBC and skip hibernate.
> Our
> > > > lives
> > > > > > will
> > > > > > > > be
> > > > > > > > > >> > > simpler,
> > > > > > > > > >> > > > especially with a schema relatively simple like
> ours
> > > > (that
> > > > > > > will
> > > > > > > > be
> > > > > > > > > >> my
> > > > > > > > > >> > > > recommendation if I was an external consultant)
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <
> > > > > > > > tzimanyi@apache.org
> > > > > > > > > >
> > > > > > > > > >> > > wrote:
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > > Hi,
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > this will be a bit off-topic. However as far as
> > > > > > > performance, I
> > > > > > > > > >> think
> > > > > > > > > >> > we
> > > > > > > > > >> > > > > should think about that we have string primary
> > keys
> > > > > > (IDs). I
> > > > > > > > > would
> > > > > > > > > >> > > expect
> > > > > > > > > >> > > > > the database systems are much better with
> indexing
> > > > > numeric
> > > > > > > > keys
> > > > > > > > > >> than
> > > > > > > > > >> > > > > strings. I remember from the past, when I was
> > > working
> > > > > with
> > > > > > > > DBs,
> > > > > > > > > >> that
> > > > > > > > > >> > > > using
> > > > > > > > > >> > > > > strings as keys or indexes was a discouraged
> > > practice.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > Best regards,
> > > > > > > > > >> > > > > Tibor
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler
> > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > >
> > > > > > > > > >> > > > > napísal(a):
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > > I changed the test to use MongoDB [1] and I
> > don't
> > > > see
> > > > > a
> > > > > > > > > >> performance
> > > > > > > > > >> > > > > > degradation with this setup [2].
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > Please keep us posted of your findings.
> Thanks!
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > Martin
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > [1]
> > > > > > > > > >> > > > >
> > > > > > > > > >> >
> > > > > > > > >
> > > > > >
> > > https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > > > > > > >> > > > > > [2]
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > ________________________________________
> > > > > > > > > >> > > > > > From: Francisco Javier Tirado Sarti <
> > > > > > ftirados@redhat.com>
> > > > > > > > > >> > > > > > Sent: Wednesday, February 7, 2024 11:40 AM
> > > > > > > > > >> > > > > > To: dev@kie.apache.org
> > > > > > > > > >> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION]
> Performance
> > > > > issues
> > > > > > > with
> > > > > > > > > >> > > data-index
> > > > > > > > > >> > > > > > persistence addon
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > yes, it can be index degradation because of
> > size,
> > > > but
> > > > > I
> > > > > > > > > believe
> > > > > > > > > >> (I
> > > > > > > > > >> > > > might
> > > > > > > > > >> > > > > be
> > > > > > > > > >> > > > > > wrong) the db is too small (yet) for that.
> > > > > > > > > >> > > > > > But, eventually, Postgres, when the DB is huge
> > > > enough,
> > > > > > > > > >> unavoidably
> > > > > > > > > >> > > will
> > > > > > > > > >> > > > > > behave like the graphic that Martin sent.
> > > > > > > > > >> > > > > > Since I believe we are not huge enough (yet),
> > lets
> > > > > rule
> > > > > > > out
> > > > > > > > > >> another
> > > > > > > > > >> > > > issue
> > > > > > > > > >> > > > > > by analysing the sql logs (I requested those
> to
> > > > Martin
> > > > > > > > offline
> > > > > > > > > >> and
> > > > > > > > > >> > he
> > > > > > > > > >> > > > is
> > > > > > > > > >> > > > > > going to kindly collect them).
> > > > > > > > > >> > > > > > Also Im curious to know if Mongo behave in the
> > > same
> > > > > way.
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique
> Gonzalez
> > > > > > Martinez <
> > > > > > > > > >> > > > > > egonzalez@apache.org> wrote:
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > > Hi Francisco,
> > > > > > > > > >> > > > > > > I would highly recommend to check indexes
> and
> > > how
> > > > > the
> > > > > > > > > updates
> > > > > > > > > >> > work
> > > > > > > > > >> > > in
> > > > > > > > > >> > > > > > data
> > > > > > > > > >> > > > > > > index to avoid full scan table and lock the
> > full
> > > > > > table.
> > > > > > > > Some
> > > > > > > > > >> db
> > > > > > > > > >> > are
> > > > > > > > > >> > > > > very
> > > > > > > > > >> > > > > > > sensitive to that.
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > El mié, 7 feb 2024, 18:41, Francisco Javier
> > > Tirado
> > > > > > > Sarti <
> > > > > > > > > >> > > > > > > ftirados@redhat.com> escribió:
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > > Hi Martin,
> > > > > > > > > >> > > > > > > > While I analyze the data, let me ask you
> if
> > it
> > > > is
> > > > > > > > possible
> > > > > > > > > >> to
> > > > > > > > > >> > > > perform
> > > > > > > > > >> > > > > > > > another check (similar in a way to
> disabling
> > > > > > > data-index
> > > > > > > > > like
> > > > > > > > > >> > you
> > > > > > > > > >> > > > do)
> > > > > > > > > >> > > > > > Can
> > > > > > > > > >> > > > > > > > you switch to MongoDB persistence and
> check
> > if
> > > > the
> > > > > > > same
> > > > > > > > > >> > > degradation
> > > > > > > > > >> > > > > > that
> > > > > > > > > >> > > > > > > is
> > > > > > > > > >> > > > > > > > there for postgres remains?
> > > > > > > > > >> > > > > > > > I do not know if this is feasible but will
> > > > > certainly
> > > > > > > > > >> indicate
> > > > > > > > > >> > the
> > > > > > > > > >> > > > > > problem
> > > > > > > > > >> > > > > > > > is on the postgres storage layer and I do
> > not
> > > > > have a
> > > > > > > > clear
> > > > > > > > > >> > > > prediction
> > > > > > > > > >> > > > > > of
> > > > > > > > > >> > > > > > > > what we will see when doing this switch.
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin
> Weiler
> > > > > > > > > >> > > > <mweiler@ibm.com.invalid
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > > > wrote:
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > > Hi Francisco,
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > thanks for your work on this important
> > > topic!
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > I would like to share some test results
> > > here,
> > > > > > which
> > > > > > > > > might
> > > > > > > > > >> > help
> > > > > > > > > >> > > to
> > > > > > > > > >> > > > > > > improve
> > > > > > > > > >> > > > > > > > > the codebase even further. I am using
> the
> > > > jmeter
> > > > > > > based
> > > > > > > > > >> test
> > > > > > > > > >> > > case
> > > > > > > > > >> > > > > from
> > > > > > > > > >> > > > > > > > Pere
> > > > > > > > > >> > > > > > > > > and Enrique (thanks guys!) [1] which
> uses
> > a
> > > > load
> > > > > > of
> > > > > > > 30
> > > > > > > > > >> > threads
> > > > > > > > > >> > > to
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > 1) start a new process instance (POST)
> > > > > > > > > >> > > > > > > > > 2) retrieve tasks for a user (GET)
> > > > > > > > > >> > > > > > > > > 3) fetches task details (GET)
> > > > > > > > > >> > > > > > > > > 4) complete a task (POST)
> > > > > > > > > >> > > > > > > > > 5) execute a query on data-audit
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > With this test setup, I noticed that the
> > > > > > performance
> > > > > > > > for
> > > > > > > > > >> the
> > > > > > > > > >> > > POST
> > > > > > > > > >> > > > > > > > > requests, in particular the one to
> start a
> > > new
> > > > > > > process
> > > > > > > > > >> > > instance,
> > > > > > > > > >> > > > > > > degrades
> > > > > > > > > >> > > > > > > > > over time - see graph [2]. If I run the
> > same
> > > > > test
> > > > > > > > > without
> > > > > > > > > >> > > > > data-index,
> > > > > > > > > >> > > > > > > > then
> > > > > > > > > >> > > > > > > > > there is no such performance degradation
> > > [3].
> > > > > You
> > > > > > > can
> > > > > > > > > >> find a
> > > > > > > > > >> > > > thread
> > > > > > > > > >> > > > > > > dump
> > > > > > > > > >> > > > > > > > > captured a few minutes into the first
> test
> > > > here
> > > > > > [4]
> > > > > > > > that
> > > > > > > > > >> > might
> > > > > > > > > >> > > > help
> > > > > > > > > >> > > > > > to
> > > > > > > > > >> > > > > > > > see
> > > > > > > > > >> > > > > > > > > some of the contention points.
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > I'd appreciate if you could take a look
> > and
> > > > see
> > > > > if
> > > > > > > > there
> > > > > > > > > >> is
> > > > > > > > > >> > > > > something
> > > > > > > > > >> > > > > > > > that
> > > > > > > > > >> > > > > > > > > can be further improved based on your
> > > previous
> > > > > > work.
> > > > > > > > If
> > > > > > > > > >> you
> > > > > > > > > >> > > need
> > > > > > > > > >> > > > > any
> > > > > > > > > >> > > > > > > > > additional data, let me know, but
> > otherwise
> > > it
> > > > > is
> > > > > > > > > >> > > straightforward
> > > > > > > > > >> > > > > to
> > > > > > > > > >> > > > > > > run
> > > > > > > > > >> > > > > > > > > the jmeter test as well.
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > Thanks,
> > > > > > > > > >> > > > > > > > > Martin
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > [1]
> > > > > > > > > >> https://github.com/pefernan/job-service-refactor-test/
> > > > > > > > > >> > > > > > > > > [2]
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > > > > > >> > > > > > > > > [3]
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > > > > > >> > > > > > > > > [4]
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > ________________________________________
> > > > > > > > > >> > > > > > > > > From: Francisco Javier Tirado Sarti <
> > > > > > > > > ftirados@redhat.com>
> > > > > > > > > >> > > > > > > > > Sent: Wednesday, January 17, 2024 9:13
> AM
> > > > > > > > > >> > > > > > > > > To: dev@kie.apache.org
> > > > > > > > > >> > > > > > > > > Cc: Pere Fernandez Perez
> > > > > > > > > >> > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION]
> > > > Performance
> > > > > > > > issues
> > > > > > > > > >> with
> > > > > > > > > >> > > > > > data-index
> > > > > > > > > >> > > > > > > > > persistence addon
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > Hi Alex,
> > > > > > > > > >> > > > > > > > > I did not take times (which depends on a
> > > > number
> > > > > of
> > > > > > > > > >> variables
> > > > > > > > > >> > > that
> > > > > > > > > >> > > > > > > > > drastically change between
> environments),
> > > but
> > > > > > verify
> > > > > > > > > that
> > > > > > > > > >> the
> > > > > > > > > >> > > > > number
> > > > > > > > > >> > > > > > of
> > > > > > > > > >> > > > > > > > > updates has been reduced drastically
> > without
> > > > > > losing
> > > > > > > > > >> > > > functionality,
> > > > > > > > > >> > > > > > > which
> > > > > > > > > >> > > > > > > > is
> > > > > > > > > >> > > > > > > > > objectively a good thing. If before the
> > > > change,
> > > > > > for
> > > > > > > > > every
> > > > > > > > > >> > node
> > > > > > > > > >> > > > > > > executed,
> > > > > > > > > >> > > > > > > > we
> > > > > > > > > >> > > > > > > > > have an update for every node previously
> > > > > executed,
> > > > > > > so
> > > > > > > > > if a
> > > > > > > > > >> > > > process
> > > > > > > > > >> > > > > > have
> > > > > > > > > >> > > > > > > > 50
> > > > > > > > > >> > > > > > > > > nodes to execute, we were performing
> > nearly
> > > > > > 50*51/2
> > > > > > > > > >> updates,
> > > > > > > > > >> > > > which
> > > > > > > > > >> > > > > > > gives
> > > > > > > > > >> > > > > > > > us
> > > > > > > > > >> > > > > > > > > a total of  1275 updates, now we have
> just
> > > one
> > > > > for
> > > > > > > > every
> > > > > > > > > >> node
> > > > > > > > > >> > > > being
> > > > > > > > > >> > > > > > > > > executed, implying a total of 50
> updates.
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex
> > > Porcelli
> > > > <
> > > > > > > > > >> > > alex@porcelli.me>
> > > > > > > > > >> > > > > > > wrote:
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > > Francisco,
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > I noticed that your PR has been
> merged,
> > > but
> > > > I
> > > > > > was
> > > > > > > > > >> expecting
> > > > > > > > > >> > > (at
> > > > > > > > > >> > > > > > least
> > > > > > > > > >> > > > > > > > > > was my understanding from this thread)
> > > that
> > > > > > before
> > > > > > > > > >> merging
> > > > > > > > > >> > > some
> > > > > > > > > >> > > > > > > > > > benchmark data would be shared in
> > advance
> > > -
> > > > to
> > > > > > > > assess
> > > > > > > > > >> the
> > > > > > > > > >> > > > > > > cost/benefit
> > > > > > > > > >> > > > > > > > > > of such a decent size change.
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > Do you have any information to share?
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM
> > Francisco
> > > > > Javier
> > > > > > > > > Tirado
> > > > > > > > > >> > Sarti
> > > > > > > > > >> > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > Yes, as intended, now we have one
> > select
> > > > and
> > > > > > one
> > > > > > > > > >> > > > insert/update
> > > > > > > > > >> > > > > > per
> > > > > > > > > >> > > > > > > > node
> > > > > > > > > >> > > > > > > > > > > event.
> > > > > > > > > >> > > > > > > > > > > I moved the PR as ready for review
> and
> > > > give
> > > > > > > @Pere
> > > > > > > > > >> > Fernandez
> > > > > > > > > >> > > > > Perez
> > > > > > > > > >> > > > > > > > > > > <pe...@redhat.com> permission to
> > the
> > > > > > branch
> > > > > > > so
> > > > > > > > > he
> > > > > > > > > >> can
> > > > > > > > > >> > > > edit
> > > > > > > > > >> > > > > it
> > > > > > > > > >> > > > > > > in
> > > > > > > > > >> > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > next two weeks (Ill be on PTO)  if
> > > > desired,
> > > > > > > before
> > > > > > > > > >> > merging.
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex
> > > > > Porcelli
> > > > > > <
> > > > > > > > > >> > > > > alex@porcelli.me>
> > > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > Cool, thank you Francisco!
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > Did you manage to get some
> > preliminary
> > > > > data
> > > > > > > > about
> > > > > > > > > >> > > > > improvements?
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM
> > > > Francisco
> > > > > > > > Javier
> > > > > > > > > >> > Tirado
> > > > > > > > > >> > > > > Sarti
> > > > > > > > > >> > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > Yes, after some delay because of
> > > > > quarkus 3
> > > > > > > > > >> migration.
> > > > > > > > > >> > > Im
> > > > > > > > > >> > > > > > > refining
> > > > > > > > > >> > > > > > > > > > this
> > > > > > > > > >> > > > > > > > > > > > > draft PR
> > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > >
> https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > On Thu, Dec 21, 2023 at 5:48 PM
> > Alex
> > > > > > > Porcelli
> > > > > > > > <
> > > > > > > > > >> > > > > > > alex@porcelli.me>
> > > > > > > > > >> > > > > > > > > > wrote:
> > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > Any update or new findings on
> > this
> > > > > > topic?
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > On Tue, Nov 28, 2023 at
> 8:38 AM
> > > > > > Francisco
> > > > > > > > > Javier
> > > > > > > > > >> > > Tirado
> > > > > > > > > >> > > > > > Sarti
> > > > > > > > > >> > > > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > Hi Alex,
> > > > > > > > > >> > > > > > > > > > > > > > > After considering different
> > > > options
> > > > > to
> > > > > > > > > improve
> > > > > > > > > >> > > > > > performance,
> > > > > > > > > >> > > > > > > > we
> > > > > > > > > >> > > > > > > > > > feel
> > > > > > > > > >> > > > > > > > > > > > that
> > > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > > > >> > > > > > > > > > > > > > > is time to "partially" move
> > away
> > > > > from
> > > > > > > the
> > > > > > > > > >> current
> > > > > > > > > >> > > Map
> > > > > > > > > >> > > > > > style
> > > > > > > > > >> > > > > > > > > > > > interface (
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > > > > >> > > > > > > > > > > > > > )
> > > > > > > > > >> > > > > > > > > > > > > > > which was shared with
> Trusty,
> > to
> > > > one
> > > > > > > more
> > > > > > > > > >> > suitable
> > > > > > > > > >> > > > for
> > > > > > > > > >> > > > > > > usage
> > > > > > > > > >> > > > > > > > > > with a
> > > > > > > > > >> > > > > > > > > > > > > > > relational DB like
> postgresql
> > > (but
> > > > > > still
> > > > > > > > > >> > compatible
> > > > > > > > > >> > > > > with
> > > > > > > > > >> > > > > > > big
> > > > > > > > > >> > > > > > > > > > table
> > > > > > > > > >> > > > > > > > > > > > dbs).
> > > > > > > > > >> > > > > > > > > > > > > > > The idea will be to replace
> > > > generic
> > > > > > > > Storage
> > > > > > > > > >> > > interface
> > > > > > > > > >> > > > > by
> > > > > > > > > >> > > > > > > four
> > > > > > > > > >> > > > > > > > > > > > specific
> > > > > > > > > >> > > > > > > > > > > > > > > interfaces (which will
> inherit
> > > > from
> > > > > a
> > > > > > > > common
> > > > > > > > > >> one
> > > > > > > > > >> > > that
> > > > > > > > > >> > > > > > keeps
> > > > > > > > > >> > > > > > > > the
> > > > > > > > > >> > > > > > > > > > query
> > > > > > > > > >> > > > > > > > > > > > > > part
> > > > > > > > > >> > > > > > > > > > > > > > > at is it. with get and query
> > > > > methods),
> > > > > > > > that
> > > > > > > > > >> will
> > > > > > > > > >> > > > > include
> > > > > > > > > >> > > > > > > the
> > > > > > > > > >> > > > > > > > > > required
> > > > > > > > > >> > > > > > > > > > > > > > > modification operations for
> > the
> > > > four
> > > > > > > > > DataIndex
> > > > > > > > > >> > > > > "domains":
> > > > > > > > > >> > > > > > > > > > > > > > processinstance,
> > > > > > > > > >> > > > > > > > > > > > > > > usertask, processdefinitions
> > and
> > > > > jobs.
> > > > > > > > Those
> > > > > > > > > >> > > > interfaces
> > > > > > > > > >> > > > > > > will
> > > > > > > > > >> > > > > > > > > > define
> > > > > > > > > >> > > > > > > > > > > > > > methods
> > > > > > > > > >> > > > > > > > > > > > > > > like addNode, addVariable,
> > > > > updateTask,
> > > > > > > > > >> > > > > addAttachment.....
> > > > > > > > > >> > > > > > > > that
> > > > > > > > > >> > > > > > > > > > will
> > > > > > > > > >> > > > > > > > > > > > allow
> > > > > > > > > >> > > > > > > > > > > > > > > the persistent layer
> > > > implementation
> > > > > > to
> > > > > > > > just
> > > > > > > > > >> > update
> > > > > > > > > >> > > > the
> > > > > > > > > >> > > > > > > > needed
> > > > > > > > > >> > > > > > > > > > info
> > > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > DB  (for example, for
> addNode
> > in
> > > > > > > Postgres,
> > > > > > > > > >> just
> > > > > > > > > >> > > > insert
> > > > > > > > > >> > > > > a
> > > > > > > > > >> > > > > > > row
> > > > > > > > > >> > > > > > > > > into
> > > > > > > > > >> > > > > > > > > > > > nodes
> > > > > > > > > >> > > > > > > > > > > > > > > table, for addNode in Mongo,
> > > > > basically
> > > > > > > the
> > > > > > > > > >> same
> > > > > > > > > >> > > > atomic
> > > > > > > > > >> > > > > > > upsert
> > > > > > > > > >> > > > > > > > > > > > operation
> > > > > > > > > >> > > > > > > > > > > > > > > that is currently done).
> > > > Therefore,
> > > > > we
> > > > > > > > > >> increase
> > > > > > > > > >> > > > > > performance
> > > > > > > > > >> > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > Postgres
> > > > > > > > > >> > > > > > > > > > > > > > > and keep the current one for
> > > > Mongo.
> > > > > > The
> > > > > > > > > >> current
> > > > > > > > > >> > DB
> > > > > > > > > >> > > > > > schemas
> > > > > > > > > >> > > > > > > > > won't
> > > > > > > > > >> > > > > > > > > > be
> > > > > > > > > >> > > > > > > > > > > > > > > touched.
> > > > > > > > > >> > > > > > > > > > > > > > > Since the code change is
> > large,
> > > I
> > > > do
> > > > > > not
> > > > > > > > > think
> > > > > > > > > >> > I'll
> > > > > > > > > >> > > > be
> > > > > > > > > >> > > > > > able
> > > > > > > > > >> > > > > > > > to
> > > > > > > > > >> > > > > > > > > > have
> > > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > PR
> > > > > > > > > >> > > > > > > > > > > > > > > ready till next week.
> > > > > > > > > >> > > > > > > > > > > > > > > But before starting, please
> > let
> > > me
> > > > > > know
> > > > > > > if
> > > > > > > > > >> that
> > > > > > > > > >> > > > > approach
> > > > > > > > > >> > > > > > is
> > > > > > > > > >> > > > > > > > > fine
> > > > > > > > > >> > > > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > you.
> > > > > > > > > >> > > > > > > > > > > > > > > Best regards.
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> > 6:55 PM
> > > > Alex
> > > > > > > > > Porcelli
> > > > > > > > > >> <
> > > > > > > > > >> > > > > > > > > alex@porcelli.me>
> > > > > > > > > >> > > > > > > > > > > > wrote:
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > Thank you Francisco to
> > getting
> > > > > > deeper
> > > > > > > on
> > > > > > > > > >> this…
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > Looking forward to see the
> > > > results
> > > > > > of
> > > > > > > > your
> > > > > > > > > >> > > > suggested
> > > > > > > > > >> > > > > > > > > > improvements.
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> > > 9:40 AM
> > > > > > > > Francisco
> > > > > > > > > >> > Javier
> > > > > > > > > >> > > > > Tirado
> > > > > > > > > >> > > > > > > > > Sarti <
> > > > > > > > > >> > > > > > > > > > > > > > > > ftirados@redhat.com>
> wrote:
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > I forgot to attach the
> > > queries
> > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> > > > 3:04 PM
> > > > > > > > > Francisco
> > > > > > > > > >> > > Javier
> > > > > > > > > >> > > > > > Tirado
> > > > > > > > > >> > > > > > > > > > Sarti <
> > > > > > > > > >> > > > > > > > > > > > > > > > > ftirados@redhat.com>
> > wrote:
> > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > >> Hi,
> > > > > > > > > >> > > > > > > > > > > > > > > > >> A brief update on this
> > > topic.
> > > > > > > > > >> > > > > > > > > > > > > > > > >> After doing a simple
> test
> > > > with
> > > > > > > > example
> > > > > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > > > >> > > > > > > > > > > > > > > > ,
> > > > > > > > > >> > > > > > > > > > > > > > > > >> the number of updates
> > over
> > > > > Nodes
> > > > > > > > table
> > > > > > > > > is
> > > > > > > > > >> > n*n,
> > > > > > > > > >> > > > so
> > > > > > > > > >> > > > > we
> > > > > > > > > >> > > > > > > > > manage
> > > > > > > > > >> > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > obtain a
> > > > > > > > > >> > > > > > > > > > > > > > > > >> perfect quadratic
> > > performance
> > > > > > > > > >> degradation.
> > > > > > > > > >> > The
> > > > > > > > > >> > > > > > problem
> > > > > > > > > >> > > > > > > > is
> > > > > > > > > >> > > > > > > > > > worse
> > > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > case
> > > > > > > > > >> > > > > > > > > > > > > > > > >> of Serverless Workflow
> > than
> > > > in
> > > > > > BPMN
> > > > > > > > > >> because
> > > > > > > > > >> > we
> > > > > > > > > >> > > > the
> > > > > > > > > >> > > > > > > > number
> > > > > > > > > >> > > > > > > > > of
> > > > > > > > > >> > > > > > > > > > > > nodes
> > > > > > > > > >> > > > > > > > > > > > > > is
> > > > > > > > > >> > > > > > > > > > > > > > > > >> greater than the number
> > of
> > > > > > states.
> > > > > > > In
> > > > > > > > > >> that
> > > > > > > > > >> > > > > example N
> > > > > > > > > >> > > > > > > is
> > > > > > > > > >> > > > > > > > > 16,
> > > > > > > > > >> > > > > > > > > > but
> > > > > > > > > >> > > > > > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > > > a
> > > > > > > > > >> > > > > > > > > > > > > > > > more
> > > > > > > > > >> > > > > > > > > > > > > > > > >> complex workflow it
> would
> > > be
> > > > > > > > certainly
> > > > > > > > > >> > large.
> > > > > > > > > >> > > > > > > > > > > > > > > > >> I think that this is
> more
> > > > > related
> > > > > > > to
> > > > > > > > > how
> > > > > > > > > >> we
> > > > > > > > > >> > > are
> > > > > > > > > >> > > > > > > handling
> > > > > > > > > >> > > > > > > > > > JPA in
> > > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > code,
> > > > > > > > > >> > > > > > > > > > > > > > > > >> in particular the
> mapping
> > > > from
> > > > > > > model
> > > > > > > > to
> > > > > > > > > >> > entity
> > > > > > > > > >> > > > > > > > (basically
> > > > > > > > > >> > > > > > > > > > JPA is
> > > > > > > > > >> > > > > > > > > > > > > > blind
> > > > > > > > > >> > > > > > > > > > > > > > > > and
> > > > > > > > > >> > > > > > > > > > > > > > > > >> has to update all nodes
> > for
> > > > > every
> > > > > > > > write
> > > > > > > > > >> > > because
> > > > > > > > > >> > > > it
> > > > > > > > > >> > > > > > > > > believes
> > > > > > > > > >> > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > node has
> > > > > > > > > >> > > > > > > > > > > > > > > > >> been updated, although
> it
> > > is
> > > > > not)
> > > > > > > > than
> > > > > > > > > an
> > > > > > > > > >> > > issue
> > > > > > > > > >> > > > in
> > > > > > > > > >> > > > > > the
> > > > > > > > > >> > > > > > > > > table
> > > > > > > > > >> > > > > > > > > > > > > > definition.
> > > > > > > > > >> > > > > > > > > > > > > > > > >> In fact, when using
> JPA,
> > > > > > separating
> > > > > > > > the
> > > > > > > > > >> > server
> > > > > > > > > >> > > > > model
> > > > > > > > > >> > > > > > > > from
> > > > > > > > > >> > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > JPA
> > > > > > > > > >> > > > > > > > > > > > > > > > entity is
> > > > > > > > > >> > > > > > > > > > > > > > > > >> not a good idea,
> > especially
> > > > if
> > > > > > the
> > > > > > > > > entity
> > > > > > > > > >> > > > contains
> > > > > > > > > >> > > > > > > > > > collections.
> > > > > > > > > >> > > > > > > > > > > > I
> > > > > > > > > >> > > > > > > > > > > > > > will
> > > > > > > > > >> > > > > > > > > > > > > > > > try
> > > > > > > > > >> > > > > > > > > > > > > > > > >> to change that without
> > > > breaking
> > > > > > > > > anything.
> > > > > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > > > > >> > > > > > > > > > > > > > > > >> On Wed, Nov 22, 2023 at
> > > > > 12:10 PM
> > > > > > > > > Enrique
> > > > > > > > > >> > > > Gonzalez
> > > > > > > > > >> > > > > > > > > Martinez <
> > > > > > > > > >> > > > > > > > > > > > > > > > >> egonzalez@apache.org>
> > > wrote:
> > > > > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> After the events split
> > you
> > > > now
> > > > > > > will
> > > > > > > > > >> need to
> > > > > > > > > >> > > > > create
> > > > > > > > > >> > > > > > a
> > > > > > > > > >> > > > > > > > node
> > > > > > > > > >> > > > > > > > > > > > instance
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> model instance of
> making
> > > > > > > independent
> > > > > > > > > >> from
> > > > > > > > > >> > the
> > > > > > > > > >> > > > > > process
> > > > > > > > > >> > > > > > > > > > instance.
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> That should do the
> > trick.
> > > > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> Regarding
> > > deleting/inserting
> > > > > it
> > > > > > > was
> > > > > > > > > >> fixed
> > > > > > > > > >> > at
> > > > > > > > > >> > > > some
> > > > > > > > > >> > > > > > > > point.
> > > > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> El mar, 21 nov 2023 a
> > las
> > > > > 20:22,
> > > > > > > > > >> Francisco
> > > > > > > > > >> > > > Javier
> > > > > > > > > >> > > > > > > > Tirado
> > > > > > > > > >> > > > > > > > > > Sarti
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> (<ftirados@redhat.com
> >)
> > > > > > escribió:
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > Hi Martin,
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > I have a task to
> > review
> > > > > > > > performance
> > > > > > > > > of
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > ProcessInstanceNodeDataEventMerger
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > My idea is to reduce
> > the
> > > > > > number
> > > > > > > of
> > > > > > > > > >> delete
> > > > > > > > > >> > > > > inserts
> > > > > > > > > >> > > > > > > > when
> > > > > > > > > >> > > > > > > > > > > > processing
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> events
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > and try to do it
> > > > > incremental.
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > That should improve
> > > > > > performance.
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > PS:
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > I was planning to
> send
> > > an
> > > > > > e-mail
> > > > > > > > > >> tomorrow
> > > > > > > > > >> > > > > > > announcing
> > > > > > > > > >> > > > > > > > > > that in
> > > > > > > > > >> > > > > > > > > > > > > > case you
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> were
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > already working on a
> > fix
> > > > for
> > > > > > > > that. I
> > > > > > > > > >> > assume
> > > > > > > > > >> > > > you
> > > > > > > > > >> > > > > > are
> > > > > > > > > >> > > > > > > > not
> > > > > > > > > >> > > > > > > > > > and I
> > > > > > > > > >> > > > > > > > > > > > > > would
> > > > > > > > > >> > > > > > > > > > > > > > > > be
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > sending a PR soon.
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > On Tue, Nov 21, 2023
> > at
> > > > > > 6:09 PM
> > > > > > > > > Martin
> > > > > > > > > >> > > Weiler
> > > > > > > > > >> > > > > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > wrote:
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > > I looked into the
> > new
> > > > > > examples
> > > > > > > > > using
> > > > > > > > > >> > > > > data-index
> > > > > > > > > >> > > > > > > > > > persistence
> > > > > > > > > >> > > > > > > > > > > > > > addon -
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> Neus'
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > > PR#1813 [1] for
> > > > serverless
> > > > > > and
> > > > > > > > > >> Pere's
> > > > > > > > > >> > > > branch
> > > > > > > > > >> > > > > > [2]
> > > > > > > > > >> > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > workflow
> > > > > > > > > >> > > > > > > > > > > > > > > > (great
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> job
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > > both!) - and they
> > work
> > > > > > without
> > > > > > > > > >> issues
> > > > > > > > > >> > > using
> > > > > > > > > >> > > > > > > single
> > > > > > > > > >> > > > > > > > > > > > requests.
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> However, under
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > > some load (I used
> > 'ab'
> > > > for
> > > > > > > > testing
> > > > > > > > > >> > with a
> > > > > > > > > >> > > > > light
> > > > > > > > > >> > > > > > > > > > > > concurrency of
> > > > > > > > > >> > > > > > > > > > > > > > 10
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> parallel
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > > requests) I ran
> into
> > > the
> > > > > > > > following
> > > > > > > > > >> > > > problems:
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > > (1) Large number
> of
> > > > > > > > insert/delete
> > > > > > > > > >> calls
> > > > > > > > > >> > > > (eg.
> > > > > > > > > >> > > > > > for
> > > > > > > > > >> > > > > > > > > tables
> > > > > > > > > >> > > > > > > > > > > > such as
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> nodes,
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > > definitions, etc)
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > > (2) Hibernate
> > > > > > > > > >> OptimisticLockExceptions
> > > > > > > > > >> > /
> > > > > > > > > >> > > > > > > > > > > > StaleStateExceptions
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > > (3) DB deadlocks
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > > (4) Error
> responses,
> > > > slow
> > > > > > > > response
> > > > > > > > > >> > times
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > > The reason I am
> > > reaching
> > > > > out
> > > > > > > > with
> > > > > > > > > >> this
> > > > > > > > > >> > > > topic
> > > > > > > > > >> > > > > > here
> > > > > > > > > >> > > > > > > > is
> > > > > > > > > >> > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > find
> > > > > > > > > >> > > > > > > > > > > > > > out if
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> we are
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > > aware of this
> issue,
> > > and
> > > > > if
> > > > > > > > > someone
> > > > > > > > > >> is
> > > > > > > > > >> > > > > already
> > > > > > > > > >> > > > > > > > > looking
> > > > > > > > > >> > > > > > > > > > > > into or
> > > > > > > > > >> > > > > > > > > > > > > > > > being
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > > assigned to it?
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > > Thanks,
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > > Martin
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > > [1]
> > > > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> >
> > > > > >
> https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > > [2]
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >>
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > > To unsubscribe,
> > > e-mail:
> > > > > > > > > >> > > > > > > > > dev-unsubscribe@kie.apache.org
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > > For additional
> > > commands,
> > > > > > > e-mail:
> > > > > > > > > >> > > > > > > > > > dev-help@kie.apache.org
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> To unsubscribe,
> e-mail:
> > > > > > > > > >> > > > > > > dev-unsubscribe@kie.apache.org
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> For additional
> commands,
> > > > > e-mail:
> > > > > > > > > >> > > > > > > > dev-help@kie.apache.org
> > > > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >>
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > > >> > > > > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > > > > > > > >> > > > > > dev-unsubscribe@kie.apache.org
> > > > > > > > > >> > > > > > > > > > > > > > > > > For additional commands,
> > > > e-mail:
> > > > > > > > > >> > > > > > > dev-help@kie.apache.org
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > > >> > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > > >
> > >
> >
>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Enrique Gonzalez Martinez <eg...@apache.org>]

Hi Francisco

The discussion we need to have before how to achieve certain features, is
the overall of the user experience. If you want to clean it up that way it
is fine by me. I have nothing against setting policy related to some sort
of clean up for a process completed some time ago.

As we discussed previously the data index is a snapshot of the last state
of the process instance included completed, but nothing was said once
completed when we need to clean that up so any policy is welcome. How to
achieve that policy is something completely different.

The main problem is how every microservice is exposing that API to the end
user or consumer (other system) and which system is going to be the façade
for complex deployments and operations. That is the discussion I was
mentioning before.

What I want to avoid is to set different policies among components and we
should strive to be as much as possible to offer certain capabilities in
the same fashion, e.g: clean up mechanism.
In the same way I want to avoid making the current system more complex than
they are. So far they are aligned offering one simple responsibility but I
would be against mixing for instance job service with data index.

El mié, 21 feb 2024 a las 12:13, Francisco Javier Tirado Sarti (<
ftirados@redhat.com>) escribió:

> Hi Enrique,
> In the case of data index I think the data to be purged is finished process
> instances (I do not think we should remove process instance that has been
> alive for ages, even if it is very likely they are not going to be ever
> completed)
> Once you delete those process instances, you also delete the associated
> user tasks and jobs.
> Therefore the problem is relatively simple, to be able to configure how
> much time a completed process instance should remain in the data index
> database. We can take a simple approach: a property with a min duration
> that cannot be changed once data index is started; a slightly complex one:
> the same property but watching it to react for changes; or the full suite:
> an admin API to be able to change the policy at any moment.
> I think this discussion is worth having.
>
> On Wed, Feb 21, 2024 at 6:15 AM Enrique Gonzalez Martinez <
> egonzalez@apache.org> wrote:
>
> > Hi Martin, the main problem regarding the purge is because it is still
> > unclear the policy for what tech to use and the future components.
> >
> > Recently we had a discussion about proposing graphql for this sort of
> admin
> > tasks. So far for subsystems we have been using rest endpoints (like
> update
> > timers or modify human task or change processes). There is one exception
> > which is the gateway that is pure graphql and somehow uses graphql for
> > everything making complex operations under the hood.
> >
> > This has somehow frozen the purge for data audit for a bit and the
> proposal
> > was to use rest endpoints to do the clean up in the component and offer
> the
> > graphql counterpart in the gateway promoting it to a first class citizen
> > component instead of having it embedded in the data index.
> >
> > I would suggest to come up at least with a policy first regarding the
> > convention every component should address this.
> >
> >
> > El mar, 20 feb 2024, 23:39, Martin Weiler <mw...@ibm.com.invalid>
> > escribió:
> >
> > > IMO, it is good to have this discussion around data sanity now instead
> of
> > > putting it off until later when data has already accumulated in
> > production
> > > environments.
> > >
> > > Based on the input here, we are dealing with three types of data:
> > > 1. Runtime data - active instances only, engine cleans up the data
> > > automatically at process instance end
> > > 2. Historic log data - data created by data-audit intended for long
> term
> > > storage
> > > 3. Data-index data - somehow this data falls in between the two
> > > aforementioned categories, with the idea of the data being "recent",
> but
> > > not restricted to active instances only
> > >
> > > We'd need purge strategies for both #2 and #3 (perhaps different ones,
> or
> > > with different config settings) in order to prevent unlimited data
> > growth.
> > >
> > > ________________________________________
> > > From: Enrique Gonzalez Martinez <eg...@apache.org>
> > > Sent: Monday, February 19, 2024 7:11 AM
> > > To: dev@kie.apache.org
> > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with data-index
> > > persistence addon
> > >
> > > Hi Francisco,
> > > To give you more context about this.
> > >
> > > STP is a concept, a process with certain constraints: no persistence
> and
> > > returning the outcome in the call (sync execution with no idle states).
> > It
> > > was a requirement from a user in the past. One of the requirements was
> > > leaving no trail. In v7 was easy because you could disable the audit in
> > > that case. Actually we have the same way to do what we did in v7 in
> here
> > as
> > > you can add/remove index just removing deps.
> > >
> > > We have the same outcome with different approaches and STP is already
> > > delivered.
> > >
> > > El lun, 19 feb 2024 a las 14:46, Francisco Javier Tirado Sarti (<
> > > ftirados@redhat.com>) escribió:
> > >
> > > > Regarding STP (which is not a concept that we have in the code. I
> mean
> > > STP
> > > > are processes as nonSTP are), I guess, as all processes, they were
> kept
> > > in
> > > > DataIndex once completed because users wanted (and still wants) to
> > check
> > > > the result once the call had been performed. If we want to leave no
> > trace
> > > > of them in DataIndex for some reason, we will need to make it a
> > > > Runtimes concept so DataIndex can handle them in a different way.
> > > >
> > > > On Mon, Feb 19, 2024 at 2:27 PM Enrique Gonzalez Martinez <
> > > > egonzalez@apache.org> wrote:
> > > >
> > > > > Alex:
> > > > > Right now the data index is working in the same way as it did in v7
> > > with
> > > > > the emitters. The only difference between two impl is that in here
> > the
> > > > > storage is pgsql instead elastic search.  You are right regarding
> is
> > a
> > > > > snapshot of the last state of the process but we did never define
> how
> > > > long
> > > > > would be alive that dats Honestly i am happy right now with the way
> > it
> > > > > works. The clean up mechanism is still tbd because we still need to
> > > > discuss
> > > > > other stuff first.
> > > > >
> > > > >
> > > > > Regarding stp is to leave no trail because u can get the outcome
> > > directly
> > > > > from the call. It was defined like that in v7. So there is no use
> for
> > > the
> > > > > index or the audit.
> > > > >
> > > > > El lun, 19 feb 2024, 14:13, Francisco Javier Tirado Sarti <
> > > > > ftirados@redhat.com> escribió:
> > > > >
> > > > > > Hi Alex,
> > > > > > There has been some confusion about the purpose of DataIndex. To
> be
> > > > > honest
> > > > > > I believe they were already sorted out, but your e-mail makes me
> > > think
> > > > > that
> > > > > > is not the case ;). I let Kris to clarify that with you. My view
> is
> > > > that
> > > > > > data-index is a way to query recently closed and active processes
> > > (the
> > > > > key
> > > > > > here is the definition of recently, which in my opinion should be
> > > > > > configurable)
> > > > > > But, besides that discussion and being pragmatic, keeping
> finishing
> > > > > process
> > > > > > instances "for a while" in DataIndex was the only way for users
> to
> > > > query
> > > > > > the result of straight through processes. That's a function that
> > > cannot
> > > > > be
> > > > > > removed right now
> > > > > >
> > > > > > On Mon, Feb 19, 2024 at 1:33 PM Alex Porcelli <
> porcelli@apache.org
> > >
> > > > > wrote:
> > > > > >
> > > > > > > if data index was supposed to provide snapshot view of the
> > process
> > > > > > > instance… why do we keep it after the process instance is
> > finished?
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Feb 19, 2024 at 7:12 AM Francisco Javier Tirado Sarti <
> > > > > > > ftirados@redhat.com> wrote:
> > > > > > >
> > > > > > > > Hi Martin.
> > > > > > > > After taking a deeper look at this, I realize that the
> > behaviour
> > > is
> > > > > the
> > > > > > > > expected one.
> > > > > > > > Runtimes DB does not track the completed process instance
> > (that's
> > > > > what
> > > > > > > the
> > > > > > > > JDBCProcessInstances warn is telling us), but DataIndex, as
> > > > expected,
> > > > > > is
> > > > > > > > tracking it in processes and nodes table. And yes it will
> grow
> > > over
> > > > > > time.
> > > > > > > > What we need is some configurable purge mechanism for
> > DataIndex,
> > > so
> > > > > it
> > > > > > > > eventually removes older completed process instances.
> > > > > > > >
> > > > > > > > On Tue, Feb 13, 2024 at 12:59 PM Francisco Javier Tirado
> Sarti
> > <
> > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > >
> > > > > > > > > Hi Martin,
> > > > > > > > > Good catch!. Looks like the skipping performed for process
> > > > > instances
> > > > > > is
> > > > > > > > > not applied to node instances. Something we definitely need
> > to
> > > > > review
> > > > > > > on
> > > > > > > > > the runtimes side.
> > > > > > > > >
> > > > > > > > > On Mon, Feb 12, 2024 at 11:59 PM Martin Weiler
> > > > > > <mweiler@ibm.com.invalid
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> On a somewhat related note, testing a simple workflow
> (start
> > > ->
> > > > > > script
> > > > > > > > >> node -> end), I see the following messages in the logs:
> > > > > > > > >> 2024-02-12 22:49:50,493 28758dde544c WARN
> > > > > > > > >> [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> > > > > > > > >> (executor-thread-3) Skipping create of process instance
> id:
> > > > > > > > >> 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2
> > > > > > > > >>
> > > > > > > > >> So far, so good. And I'd expect to see no trace of this
> > > process
> > > > in
> > > > > > the
> > > > > > > > >> database if I don't have data audit enabled.
> > > > > > > > >>
> > > > > > > > >> However, the 'processes' table contains a row with
> state=2,
> > > with
> > > > > > > related
> > > > > > > > >> entries in the 'nodes' table. In a load test, I see these
> > > tables
> > > > > > grow
> > > > > > > > >> significantly over time. Am I missing something to have
> > these
> > > > > > entries
> > > > > > > > >> cleaned up automatically?
> > > > > > > > >>
> > > > > > > > >> ________________________________________
> > > > > > > > >> From: Martin Weiler <mw...@ibm.com.INVALID>
> > > > > > > > >> Sent: Monday, February 12, 2024 3:40 PM
> > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > >> Subject: [EXTERNAL] RE: [DISCUSSION] Performance issues
> with
> > > > > > > data-index
> > > > > > > > >> persistence addon
> > > > > > > > >>
> > > > > > > > >> Thanks everyone for your input. Based on this discussion,
> I
> > > > opened
> > > > > > the
> > > > > > > > >> following PR:
> > > > > > > > >>
> > https://github.com/apache/incubator-kie-kogito-apps/pull/1985
> > > > > > > > >>
> > > > > > > > >> With this change, the performance seems to be stable over
> > > time:
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
> > > > > > > > >>
> > > > > > > > >> Martin
> > > > > > > > >>
> > > > > > > > >> ________________________________________
> > > > > > > > >> From: Gonzalo Muñoz <gm...@apache.org>
> > > > > > > > >> Sent: Friday, February 9, 2024 9:42 AM
> > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > >> Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues
> with
> > > > > > > data-index
> > > > > > > > >> persistence addon
> > > > > > > > >>
> > > > > > > > >> Great work Francisco,
> > > > > > > > >> Martin, take a look at this link with some related tips
> (in
> > > case
> > > > > you
> > > > > > > > find
> > > > > > > > >> it useful):
> > > > > > > > >>
> > > https://www.cybertec-postgresql.com/en/index-your-foreign-key/
> > > > > > > > >>
> > > > > > > > >> El vie, 9 feb 2024 a las 17:20, Francisco Javier Tirado
> > Sarti
> > > (<
> > > > > > > > >> ftirados@redhat.com>) escribió:
> > > > > > > > >>
> > > > > > > > >> > For the moment being, we will keep JPA till we exhaust
> all
> > > > > > > > >> possibilities,
> > > > > > > > >> > let's call switching from jpa to jdbc our hidden plan B
> ;)
> > > > > > > > >> > I already told Martin, but in order everyone to know,
> just
> > > > after
> > > > > > > > writing
> > > > > > > > >> > the previous email, I thought "what if Postgres is not
> > > > > > automatically
> > > > > > > > >> > indexing foreign keys like mysql?" and, eureka
> > > > > > > > >> > Postgres doc
> > > > > > > > >> >
> > > https://www.postgresql.org/docs/current/ddl-constraints.html
> > > > > > > > >> > Mysql doc
> > > > > > > > >> >
> > > > > >
> > https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> > > > > > > > >> > These are the relevant excerpt
> > > > > > > > >> >
> > > > > > > > >> > *Postgresql*
> > > > > > > > >> > *A foreign key must reference columns that either are a
> > > > primary
> > > > > > key
> > > > > > > or
> > > > > > > > >> form
> > > > > > > > >> > a unique constraint, or are columns from a non-partial
> > > unique
> > > > > > index.
> > > > > > > > >> This
> > > > > > > > >> > means that the referenced columns always have an index
> to
> > > > allow
> > > > > > > > >> efficient
> > > > > > > > >> > lookups on whether a referencing row has a match. Since
> a
> > > > DELETE
> > > > > > of
> > > > > > > a
> > > > > > > > >> row
> > > > > > > > >> > from the referenced table or an UPDATE of a referenced
> > > column
> > > > > will
> > > > > > > > >> require
> > > > > > > > >> > a scan of the referencing table for rows matching the
> old
> > > > value,
> > > > > > it
> > > > > > > is
> > > > > > > > >> > often a good idea to index the referencing columns too.
> > > > Because
> > > > > > this
> > > > > > > > is
> > > > > > > > >> not
> > > > > > > > >> > always needed, and there are many choices available on
> how
> > > to
> > > > > > index,
> > > > > > > > the
> > > > > > > > >> > declaration of a foreign key constraint does not
> > > automatically
> > > > > > > create
> > > > > > > > an
> > > > > > > > >> > index on the referencing columns.*
> > > > > > > > >> > *Mysql*
> > > > > > > > >> > *MySQL requires that foreign key columns be indexed; if
> > you
> > > > > > create a
> > > > > > > > >> table
> > > > > > > > >> > with a foreign key constraint but no index on a given
> > > column,
> > > > an
> > > > > > > index
> > > > > > > > >> is
> > > > > > > > >> > created. *
> > > > > > > > >> >
> > > > > > > > >> > So I asked Martin to especially create an index for
> > > > > > > > process_instance_id
> > > > > > > > >> > column on nodes table
> > > > > > > > >> > I think that will fix the problem detected on the thread
> > > dump.
> > > > > > > > >> > The simpler process test to verify queries are fine
> still
> > > > > stands,
> > > > > > > > >> though ;)
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <
> > > > > tzimanyi@apache.org
> > > > > > >
> > > > > > > > >> wrote:
> > > > > > > > >> >
> > > > > > > > >> > > I always preferred pure JDBC over Hibernate myself,
> just
> > > for
> > > > > the
> > > > > > > > sake
> > > > > > > > >> of
> > > > > > > > >> > > control of what is happening :) So I would not -1 that
> > > > myself.
> > > > > > > > >> > >
> > > > > > > > >> > > Tibor
> > > > > > > > >> > >
> > > > > > > > >> > > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado
> Sarti <
> > > > > > > > >> > > ftirados@redhat.com>
> > > > > > > > >> > > napísal(a):
> > > > > > > > >> > >
> > > > > > > > >> > > > Hi,
> > > > > > > > >> > > > Usually I do not want to talk about work in progress
> > > > because
> > > > > > > > >> > preliminary
> > > > > > > > >> > > > conclusions are pretty volatile but, well, there
> are a
> > > > > couple
> > > > > > of
> > > > > > > > >> things
> > > > > > > > >> > > > that can be concluded from the really valuable
> > > information
> > > > > > that
> > > > > > > > >> Martin
> > > > > > > > >> > > > provided.
> > > > > > > > >> > > > 1) In order to be able to determine if the number of
> > > > > > statements
> > > > > > > is
> > > > > > > > >> > larger
> > > > > > > > >> > > > than expected, I asked Martin to test with a simpler
> > > > process
> > > > > > > > >> > definition.
> > > > > > > > >> > > > One with just three nodes: start, script and end.
> The
> > > > script
> > > > > > one
> > > > > > > > >> should
> > > > > > > > >> > > > change just one variable. This way we can analyze if
> > the
> > > > > > number
> > > > > > > of
> > > > > > > > >> > > queries
> > > > > > > > >> > > > is the expected one. From the single log (audit was
> > > > > activated
> > > > > > > > them)
> > > > > > > > >> my
> > > > > > > > >> > > > conclusion is that the number of insert/updates over
> > > > > processes
> > > > > > > and
> > > > > > > > >> > nodes
> > > > > > > > >> > > > (there a lot over task, that I will prefer to skip
> for
> > > > now,
> > > > > > baby
> > > > > > > > >> steps)
> > > > > > > > >> > > is
> > > > > > > > >> > > > the expected one.
> > > > > > > > >> > > > 2) Analysing the thread dump, we see around 15
> threads
> > > > > > executing
> > > > > > > > >> this
> > > > > > > > >> > > line
> > > > > > > > >> > > > at
> > > > > > > > >> > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > > > > > > >> > > > so its pretty clear the code to be optimized ;). I'm
> > > > > > evaluating
> > > > > > > > >> > > > possibilities within JPA/Hibernate, but I'm starting
> > to
> > > > > think
> > > > > > > that
> > > > > > > > >> it
> > > > > > > > >> > > might
> > > > > > > > >> > > > be better to switch to JDBC and skip hibernate. Our
> > > lives
> > > > > will
> > > > > > > be
> > > > > > > > >> > > simpler,
> > > > > > > > >> > > > especially with a schema relatively simple like ours
> > > (that
> > > > > > will
> > > > > > > be
> > > > > > > > >> my
> > > > > > > > >> > > > recommendation if I was an external consultant)
> > > > > > > > >> > > >
> > > > > > > > >> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <
> > > > > > > tzimanyi@apache.org
> > > > > > > > >
> > > > > > > > >> > > wrote:
> > > > > > > > >> > > >
> > > > > > > > >> > > > > Hi,
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > this will be a bit off-topic. However as far as
> > > > > > performance, I
> > > > > > > > >> think
> > > > > > > > >> > we
> > > > > > > > >> > > > > should think about that we have string primary
> keys
> > > > > (IDs). I
> > > > > > > > would
> > > > > > > > >> > > expect
> > > > > > > > >> > > > > the database systems are much better with indexing
> > > > numeric
> > > > > > > keys
> > > > > > > > >> than
> > > > > > > > >> > > > > strings. I remember from the past, when I was
> > working
> > > > with
> > > > > > > DBs,
> > > > > > > > >> that
> > > > > > > > >> > > > using
> > > > > > > > >> > > > > strings as keys or indexes was a discouraged
> > practice.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Best regards,
> > > > > > > > >> > > > > Tibor
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler
> > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > >
> > > > > > > > >> > > > > napísal(a):
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > > I changed the test to use MongoDB [1] and I
> don't
> > > see
> > > > a
> > > > > > > > >> performance
> > > > > > > > >> > > > > > degradation with this setup [2].
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > Please keep us posted of your findings. Thanks!
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > Martin
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > [1]
> > > > > > > > >> > > > >
> > > > > > > > >> >
> > > > > > > >
> > > > >
> > https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > > > > > >> > > > > > [2]
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > ________________________________________
> > > > > > > > >> > > > > > From: Francisco Javier Tirado Sarti <
> > > > > ftirados@redhat.com>
> > > > > > > > >> > > > > > Sent: Wednesday, February 7, 2024 11:40 AM
> > > > > > > > >> > > > > > To: dev@kie.apache.org
> > > > > > > > >> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance
> > > > issues
> > > > > > with
> > > > > > > > >> > > data-index
> > > > > > > > >> > > > > > persistence addon
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > yes, it can be index degradation because of
> size,
> > > but
> > > > I
> > > > > > > > believe
> > > > > > > > >> (I
> > > > > > > > >> > > > might
> > > > > > > > >> > > > > be
> > > > > > > > >> > > > > > wrong) the db is too small (yet) for that.
> > > > > > > > >> > > > > > But, eventually, Postgres, when the DB is huge
> > > enough,
> > > > > > > > >> unavoidably
> > > > > > > > >> > > will
> > > > > > > > >> > > > > > behave like the graphic that Martin sent.
> > > > > > > > >> > > > > > Since I believe we are not huge enough (yet),
> lets
> > > > rule
> > > > > > out
> > > > > > > > >> another
> > > > > > > > >> > > > issue
> > > > > > > > >> > > > > > by analysing the sql logs (I requested those to
> > > Martin
> > > > > > > offline
> > > > > > > > >> and
> > > > > > > > >> > he
> > > > > > > > >> > > > is
> > > > > > > > >> > > > > > going to kindly collect them).
> > > > > > > > >> > > > > > Also Im curious to know if Mongo behave in the
> > same
> > > > way.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez
> > > > > Martinez <
> > > > > > > > >> > > > > > egonzalez@apache.org> wrote:
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > > Hi Francisco,
> > > > > > > > >> > > > > > > I would highly recommend to check indexes and
> > how
> > > > the
> > > > > > > > updates
> > > > > > > > >> > work
> > > > > > > > >> > > in
> > > > > > > > >> > > > > > data
> > > > > > > > >> > > > > > > index to avoid full scan table and lock the
> full
> > > > > table.
> > > > > > > Some
> > > > > > > > >> db
> > > > > > > > >> > are
> > > > > > > > >> > > > > very
> > > > > > > > >> > > > > > > sensitive to that.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > El mié, 7 feb 2024, 18:41, Francisco Javier
> > Tirado
> > > > > > Sarti <
> > > > > > > > >> > > > > > > ftirados@redhat.com> escribió:
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > > Hi Martin,
> > > > > > > > >> > > > > > > > While I analyze the data, let me ask you if
> it
> > > is
> > > > > > > possible
> > > > > > > > >> to
> > > > > > > > >> > > > perform
> > > > > > > > >> > > > > > > > another check (similar in a way to disabling
> > > > > > data-index
> > > > > > > > like
> > > > > > > > >> > you
> > > > > > > > >> > > > do)
> > > > > > > > >> > > > > > Can
> > > > > > > > >> > > > > > > > you switch to MongoDB persistence and check
> if
> > > the
> > > > > > same
> > > > > > > > >> > > degradation
> > > > > > > > >> > > > > > that
> > > > > > > > >> > > > > > > is
> > > > > > > > >> > > > > > > > there for postgres remains?
> > > > > > > > >> > > > > > > > I do not know if this is feasible but will
> > > > certainly
> > > > > > > > >> indicate
> > > > > > > > >> > the
> > > > > > > > >> > > > > > problem
> > > > > > > > >> > > > > > > > is on the postgres storage layer and I do
> not
> > > > have a
> > > > > > > clear
> > > > > > > > >> > > > prediction
> > > > > > > > >> > > > > > of
> > > > > > > > >> > > > > > > > what we will see when doing this switch.
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler
> > > > > > > > >> > > > <mweiler@ibm.com.invalid
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > > > wrote:
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > > Hi Francisco,
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > thanks for your work on this important
> > topic!
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > I would like to share some test results
> > here,
> > > > > which
> > > > > > > > might
> > > > > > > > >> > help
> > > > > > > > >> > > to
> > > > > > > > >> > > > > > > improve
> > > > > > > > >> > > > > > > > > the codebase even further. I am using the
> > > jmeter
> > > > > > based
> > > > > > > > >> test
> > > > > > > > >> > > case
> > > > > > > > >> > > > > from
> > > > > > > > >> > > > > > > > Pere
> > > > > > > > >> > > > > > > > > and Enrique (thanks guys!) [1] which uses
> a
> > > load
> > > > > of
> > > > > > 30
> > > > > > > > >> > threads
> > > > > > > > >> > > to
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > 1) start a new process instance (POST)
> > > > > > > > >> > > > > > > > > 2) retrieve tasks for a user (GET)
> > > > > > > > >> > > > > > > > > 3) fetches task details (GET)
> > > > > > > > >> > > > > > > > > 4) complete a task (POST)
> > > > > > > > >> > > > > > > > > 5) execute a query on data-audit
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > With this test setup, I noticed that the
> > > > > performance
> > > > > > > for
> > > > > > > > >> the
> > > > > > > > >> > > POST
> > > > > > > > >> > > > > > > > > requests, in particular the one to start a
> > new
> > > > > > process
> > > > > > > > >> > > instance,
> > > > > > > > >> > > > > > > degrades
> > > > > > > > >> > > > > > > > > over time - see graph [2]. If I run the
> same
> > > > test
> > > > > > > > without
> > > > > > > > >> > > > > data-index,
> > > > > > > > >> > > > > > > > then
> > > > > > > > >> > > > > > > > > there is no such performance degradation
> > [3].
> > > > You
> > > > > > can
> > > > > > > > >> find a
> > > > > > > > >> > > > thread
> > > > > > > > >> > > > > > > dump
> > > > > > > > >> > > > > > > > > captured a few minutes into the first test
> > > here
> > > > > [4]
> > > > > > > that
> > > > > > > > >> > might
> > > > > > > > >> > > > help
> > > > > > > > >> > > > > > to
> > > > > > > > >> > > > > > > > see
> > > > > > > > >> > > > > > > > > some of the contention points.
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > I'd appreciate if you could take a look
> and
> > > see
> > > > if
> > > > > > > there
> > > > > > > > >> is
> > > > > > > > >> > > > > something
> > > > > > > > >> > > > > > > > that
> > > > > > > > >> > > > > > > > > can be further improved based on your
> > previous
> > > > > work.
> > > > > > > If
> > > > > > > > >> you
> > > > > > > > >> > > need
> > > > > > > > >> > > > > any
> > > > > > > > >> > > > > > > > > additional data, let me know, but
> otherwise
> > it
> > > > is
> > > > > > > > >> > > straightforward
> > > > > > > > >> > > > > to
> > > > > > > > >> > > > > > > run
> > > > > > > > >> > > > > > > > > the jmeter test as well.
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > Thanks,
> > > > > > > > >> > > > > > > > > Martin
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > [1]
> > > > > > > > >> https://github.com/pefernan/job-service-refactor-test/
> > > > > > > > >> > > > > > > > > [2]
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > > > > >> > > > > > > > > [3]
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > > > > >> > > > > > > > > [4]
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > ________________________________________
> > > > > > > > >> > > > > > > > > From: Francisco Javier Tirado Sarti <
> > > > > > > > ftirados@redhat.com>
> > > > > > > > >> > > > > > > > > Sent: Wednesday, January 17, 2024 9:13 AM
> > > > > > > > >> > > > > > > > > To: dev@kie.apache.org
> > > > > > > > >> > > > > > > > > Cc: Pere Fernandez Perez
> > > > > > > > >> > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION]
> > > Performance
> > > > > > > issues
> > > > > > > > >> with
> > > > > > > > >> > > > > > data-index
> > > > > > > > >> > > > > > > > > persistence addon
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > Hi Alex,
> > > > > > > > >> > > > > > > > > I did not take times (which depends on a
> > > number
> > > > of
> > > > > > > > >> variables
> > > > > > > > >> > > that
> > > > > > > > >> > > > > > > > > drastically change between environments),
> > but
> > > > > verify
> > > > > > > > that
> > > > > > > > >> the
> > > > > > > > >> > > > > number
> > > > > > > > >> > > > > > of
> > > > > > > > >> > > > > > > > > updates has been reduced drastically
> without
> > > > > losing
> > > > > > > > >> > > > functionality,
> > > > > > > > >> > > > > > > which
> > > > > > > > >> > > > > > > > is
> > > > > > > > >> > > > > > > > > objectively a good thing. If before the
> > > change,
> > > > > for
> > > > > > > > every
> > > > > > > > >> > node
> > > > > > > > >> > > > > > > executed,
> > > > > > > > >> > > > > > > > we
> > > > > > > > >> > > > > > > > > have an update for every node previously
> > > > executed,
> > > > > > so
> > > > > > > > if a
> > > > > > > > >> > > > process
> > > > > > > > >> > > > > > have
> > > > > > > > >> > > > > > > > 50
> > > > > > > > >> > > > > > > > > nodes to execute, we were performing
> nearly
> > > > > 50*51/2
> > > > > > > > >> updates,
> > > > > > > > >> > > > which
> > > > > > > > >> > > > > > > gives
> > > > > > > > >> > > > > > > > us
> > > > > > > > >> > > > > > > > > a total of  1275 updates, now we have just
> > one
> > > > for
> > > > > > > every
> > > > > > > > >> node
> > > > > > > > >> > > > being
> > > > > > > > >> > > > > > > > > executed, implying a total of 50 updates.
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex
> > Porcelli
> > > <
> > > > > > > > >> > > alex@porcelli.me>
> > > > > > > > >> > > > > > > wrote:
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > > Francisco,
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > I noticed that your PR has been merged,
> > but
> > > I
> > > > > was
> > > > > > > > >> expecting
> > > > > > > > >> > > (at
> > > > > > > > >> > > > > > least
> > > > > > > > >> > > > > > > > > > was my understanding from this thread)
> > that
> > > > > before
> > > > > > > > >> merging
> > > > > > > > >> > > some
> > > > > > > > >> > > > > > > > > > benchmark data would be shared in
> advance
> > -
> > > to
> > > > > > > assess
> > > > > > > > >> the
> > > > > > > > >> > > > > > > cost/benefit
> > > > > > > > >> > > > > > > > > > of such a decent size change.
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > Do you have any information to share?
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM
> Francisco
> > > > Javier
> > > > > > > > Tirado
> > > > > > > > >> > Sarti
> > > > > > > > >> > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > Yes, as intended, now we have one
> select
> > > and
> > > > > one
> > > > > > > > >> > > > insert/update
> > > > > > > > >> > > > > > per
> > > > > > > > >> > > > > > > > node
> > > > > > > > >> > > > > > > > > > > event.
> > > > > > > > >> > > > > > > > > > > I moved the PR as ready for review and
> > > give
> > > > > > @Pere
> > > > > > > > >> > Fernandez
> > > > > > > > >> > > > > Perez
> > > > > > > > >> > > > > > > > > > > <pe...@redhat.com> permission to
> the
> > > > > branch
> > > > > > so
> > > > > > > > he
> > > > > > > > >> can
> > > > > > > > >> > > > edit
> > > > > > > > >> > > > > it
> > > > > > > > >> > > > > > > in
> > > > > > > > >> > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > next two weeks (Ill be on PTO)  if
> > > desired,
> > > > > > before
> > > > > > > > >> > merging.
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex
> > > > Porcelli
> > > > > <
> > > > > > > > >> > > > > alex@porcelli.me>
> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > Cool, thank you Francisco!
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > Did you manage to get some
> preliminary
> > > > data
> > > > > > > about
> > > > > > > > >> > > > > improvements?
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM
> > > Francisco
> > > > > > > Javier
> > > > > > > > >> > Tirado
> > > > > > > > >> > > > > Sarti
> > > > > > > > >> > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > Yes, after some delay because of
> > > > quarkus 3
> > > > > > > > >> migration.
> > > > > > > > >> > > Im
> > > > > > > > >> > > > > > > refining
> > > > > > > > >> > > > > > > > > > this
> > > > > > > > >> > > > > > > > > > > > > draft PR
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > On Thu, Dec 21, 2023 at 5:48 PM
> Alex
> > > > > > Porcelli
> > > > > > > <
> > > > > > > > >> > > > > > > alex@porcelli.me>
> > > > > > > > >> > > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > Any update or new findings on
> this
> > > > > topic?
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM
> > > > > Francisco
> > > > > > > > Javier
> > > > > > > > >> > > Tirado
> > > > > > > > >> > > > > > Sarti
> > > > > > > > >> > > > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > Hi Alex,
> > > > > > > > >> > > > > > > > > > > > > > > After considering different
> > > options
> > > > to
> > > > > > > > improve
> > > > > > > > >> > > > > > performance,
> > > > > > > > >> > > > > > > > we
> > > > > > > > >> > > > > > > > > > feel
> > > > > > > > >> > > > > > > > > > > > that
> > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > > >> > > > > > > > > > > > > > > is time to "partially" move
> away
> > > > from
> > > > > > the
> > > > > > > > >> current
> > > > > > > > >> > > Map
> > > > > > > > >> > > > > > style
> > > > > > > > >> > > > > > > > > > > > interface (
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > > > >> > > > > > > > > > > > > > )
> > > > > > > > >> > > > > > > > > > > > > > > which was shared with Trusty,
> to
> > > one
> > > > > > more
> > > > > > > > >> > suitable
> > > > > > > > >> > > > for
> > > > > > > > >> > > > > > > usage
> > > > > > > > >> > > > > > > > > > with a
> > > > > > > > >> > > > > > > > > > > > > > > relational DB like postgresql
> > (but
> > > > > still
> > > > > > > > >> > compatible
> > > > > > > > >> > > > > with
> > > > > > > > >> > > > > > > big
> > > > > > > > >> > > > > > > > > > table
> > > > > > > > >> > > > > > > > > > > > dbs).
> > > > > > > > >> > > > > > > > > > > > > > > The idea will be to replace
> > > generic
> > > > > > > Storage
> > > > > > > > >> > > interface
> > > > > > > > >> > > > > by
> > > > > > > > >> > > > > > > four
> > > > > > > > >> > > > > > > > > > > > specific
> > > > > > > > >> > > > > > > > > > > > > > > interfaces (which will inherit
> > > from
> > > > a
> > > > > > > common
> > > > > > > > >> one
> > > > > > > > >> > > that
> > > > > > > > >> > > > > > keeps
> > > > > > > > >> > > > > > > > the
> > > > > > > > >> > > > > > > > > > query
> > > > > > > > >> > > > > > > > > > > > > > part
> > > > > > > > >> > > > > > > > > > > > > > > at is it. with get and query
> > > > methods),
> > > > > > > that
> > > > > > > > >> will
> > > > > > > > >> > > > > include
> > > > > > > > >> > > > > > > the
> > > > > > > > >> > > > > > > > > > required
> > > > > > > > >> > > > > > > > > > > > > > > modification operations for
> the
> > > four
> > > > > > > > DataIndex
> > > > > > > > >> > > > > "domains":
> > > > > > > > >> > > > > > > > > > > > > > processinstance,
> > > > > > > > >> > > > > > > > > > > > > > > usertask, processdefinitions
> and
> > > > jobs.
> > > > > > > Those
> > > > > > > > >> > > > interfaces
> > > > > > > > >> > > > > > > will
> > > > > > > > >> > > > > > > > > > define
> > > > > > > > >> > > > > > > > > > > > > > methods
> > > > > > > > >> > > > > > > > > > > > > > > like addNode, addVariable,
> > > > updateTask,
> > > > > > > > >> > > > > addAttachment.....
> > > > > > > > >> > > > > > > > that
> > > > > > > > >> > > > > > > > > > will
> > > > > > > > >> > > > > > > > > > > > allow
> > > > > > > > >> > > > > > > > > > > > > > > the persistent layer
> > > implementation
> > > > > to
> > > > > > > just
> > > > > > > > >> > update
> > > > > > > > >> > > > the
> > > > > > > > >> > > > > > > > needed
> > > > > > > > >> > > > > > > > > > info
> > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > DB  (for example, for addNode
> in
> > > > > > Postgres,
> > > > > > > > >> just
> > > > > > > > >> > > > insert
> > > > > > > > >> > > > > a
> > > > > > > > >> > > > > > > row
> > > > > > > > >> > > > > > > > > into
> > > > > > > > >> > > > > > > > > > > > nodes
> > > > > > > > >> > > > > > > > > > > > > > > table, for addNode in Mongo,
> > > > basically
> > > > > > the
> > > > > > > > >> same
> > > > > > > > >> > > > atomic
> > > > > > > > >> > > > > > > upsert
> > > > > > > > >> > > > > > > > > > > > operation
> > > > > > > > >> > > > > > > > > > > > > > > that is currently done).
> > > Therefore,
> > > > we
> > > > > > > > >> increase
> > > > > > > > >> > > > > > performance
> > > > > > > > >> > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > Postgres
> > > > > > > > >> > > > > > > > > > > > > > > and keep the current one for
> > > Mongo.
> > > > > The
> > > > > > > > >> current
> > > > > > > > >> > DB
> > > > > > > > >> > > > > > schemas
> > > > > > > > >> > > > > > > > > won't
> > > > > > > > >> > > > > > > > > > be
> > > > > > > > >> > > > > > > > > > > > > > > touched.
> > > > > > > > >> > > > > > > > > > > > > > > Since the code change is
> large,
> > I
> > > do
> > > > > not
> > > > > > > > think
> > > > > > > > >> > I'll
> > > > > > > > >> > > > be
> > > > > > > > >> > > > > > able
> > > > > > > > >> > > > > > > > to
> > > > > > > > >> > > > > > > > > > have
> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > PR
> > > > > > > > >> > > > > > > > > > > > > > > ready till next week.
> > > > > > > > >> > > > > > > > > > > > > > > But before starting, please
> let
> > me
> > > > > know
> > > > > > if
> > > > > > > > >> that
> > > > > > > > >> > > > > approach
> > > > > > > > >> > > > > > is
> > > > > > > > >> > > > > > > > > fine
> > > > > > > > >> > > > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > you.
> > > > > > > > >> > > > > > > > > > > > > > > Best regards.
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> 6:55 PM
> > > Alex
> > > > > > > > Porcelli
> > > > > > > > >> <
> > > > > > > > >> > > > > > > > > alex@porcelli.me>
> > > > > > > > >> > > > > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > Thank you Francisco to
> getting
> > > > > deeper
> > > > > > on
> > > > > > > > >> this…
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > Looking forward to see the
> > > results
> > > > > of
> > > > > > > your
> > > > > > > > >> > > > suggested
> > > > > > > > >> > > > > > > > > > improvements.
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> > 9:40 AM
> > > > > > > Francisco
> > > > > > > > >> > Javier
> > > > > > > > >> > > > > Tirado
> > > > > > > > >> > > > > > > > > Sarti <
> > > > > > > > >> > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > I forgot to attach the
> > queries
> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> > > 3:04 PM
> > > > > > > > Francisco
> > > > > > > > >> > > Javier
> > > > > > > > >> > > > > > Tirado
> > > > > > > > >> > > > > > > > > > Sarti <
> > > > > > > > >> > > > > > > > > > > > > > > > > ftirados@redhat.com>
> wrote:
> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > >> Hi,
> > > > > > > > >> > > > > > > > > > > > > > > > >> A brief update on this
> > topic.
> > > > > > > > >> > > > > > > > > > > > > > > > >> After doing a simple test
> > > with
> > > > > > > example
> > > > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > > >> > > > > > > > > > > > > > > > ,
> > > > > > > > >> > > > > > > > > > > > > > > > >> the number of updates
> over
> > > > Nodes
> > > > > > > table
> > > > > > > > is
> > > > > > > > >> > n*n,
> > > > > > > > >> > > > so
> > > > > > > > >> > > > > we
> > > > > > > > >> > > > > > > > > manage
> > > > > > > > >> > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > obtain a
> > > > > > > > >> > > > > > > > > > > > > > > > >> perfect quadratic
> > performance
> > > > > > > > >> degradation.
> > > > > > > > >> > The
> > > > > > > > >> > > > > > problem
> > > > > > > > >> > > > > > > > is
> > > > > > > > >> > > > > > > > > > worse
> > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > case
> > > > > > > > >> > > > > > > > > > > > > > > > >> of Serverless Workflow
> than
> > > in
> > > > > BPMN
> > > > > > > > >> because
> > > > > > > > >> > we
> > > > > > > > >> > > > the
> > > > > > > > >> > > > > > > > number
> > > > > > > > >> > > > > > > > > of
> > > > > > > > >> > > > > > > > > > > > nodes
> > > > > > > > >> > > > > > > > > > > > > > is
> > > > > > > > >> > > > > > > > > > > > > > > > >> greater than the number
> of
> > > > > states.
> > > > > > In
> > > > > > > > >> that
> > > > > > > > >> > > > > example N
> > > > > > > > >> > > > > > > is
> > > > > > > > >> > > > > > > > > 16,
> > > > > > > > >> > > > > > > > > > but
> > > > > > > > >> > > > > > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > > > a
> > > > > > > > >> > > > > > > > > > > > > > > > more
> > > > > > > > >> > > > > > > > > > > > > > > > >> complex workflow it would
> > be
> > > > > > > certainly
> > > > > > > > >> > large.
> > > > > > > > >> > > > > > > > > > > > > > > > >> I think that this is more
> > > > related
> > > > > > to
> > > > > > > > how
> > > > > > > > >> we
> > > > > > > > >> > > are
> > > > > > > > >> > > > > > > handling
> > > > > > > > >> > > > > > > > > > JPA in
> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > code,
> > > > > > > > >> > > > > > > > > > > > > > > > >> in particular the mapping
> > > from
> > > > > > model
> > > > > > > to
> > > > > > > > >> > entity
> > > > > > > > >> > > > > > > > (basically
> > > > > > > > >> > > > > > > > > > JPA is
> > > > > > > > >> > > > > > > > > > > > > > blind
> > > > > > > > >> > > > > > > > > > > > > > > > and
> > > > > > > > >> > > > > > > > > > > > > > > > >> has to update all nodes
> for
> > > > every
> > > > > > > write
> > > > > > > > >> > > because
> > > > > > > > >> > > > it
> > > > > > > > >> > > > > > > > > believes
> > > > > > > > >> > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > node has
> > > > > > > > >> > > > > > > > > > > > > > > > >> been updated, although it
> > is
> > > > not)
> > > > > > > than
> > > > > > > > an
> > > > > > > > >> > > issue
> > > > > > > > >> > > > in
> > > > > > > > >> > > > > > the
> > > > > > > > >> > > > > > > > > table
> > > > > > > > >> > > > > > > > > > > > > > definition.
> > > > > > > > >> > > > > > > > > > > > > > > > >> In fact, when using JPA,
> > > > > separating
> > > > > > > the
> > > > > > > > >> > server
> > > > > > > > >> > > > > model
> > > > > > > > >> > > > > > > > from
> > > > > > > > >> > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > JPA
> > > > > > > > >> > > > > > > > > > > > > > > > entity is
> > > > > > > > >> > > > > > > > > > > > > > > > >> not a good idea,
> especially
> > > if
> > > > > the
> > > > > > > > entity
> > > > > > > > >> > > > contains
> > > > > > > > >> > > > > > > > > > collections.
> > > > > > > > >> > > > > > > > > > > > I
> > > > > > > > >> > > > > > > > > > > > > > will
> > > > > > > > >> > > > > > > > > > > > > > > > try
> > > > > > > > >> > > > > > > > > > > > > > > > >> to change that without
> > > breaking
> > > > > > > > anything.
> > > > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > > > > > > > > >> On Wed, Nov 22, 2023 at
> > > > 12:10 PM
> > > > > > > > Enrique
> > > > > > > > >> > > > Gonzalez
> > > > > > > > >> > > > > > > > > Martinez <
> > > > > > > > >> > > > > > > > > > > > > > > > >> egonzalez@apache.org>
> > wrote:
> > > > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > > > > > > > > >>> After the events split
> you
> > > now
> > > > > > will
> > > > > > > > >> need to
> > > > > > > > >> > > > > create
> > > > > > > > >> > > > > > a
> > > > > > > > >> > > > > > > > node
> > > > > > > > >> > > > > > > > > > > > instance
> > > > > > > > >> > > > > > > > > > > > > > > > >>> model instance of making
> > > > > > independent
> > > > > > > > >> from
> > > > > > > > >> > the
> > > > > > > > >> > > > > > process
> > > > > > > > >> > > > > > > > > > instance.
> > > > > > > > >> > > > > > > > > > > > > > > > >>> That should do the
> trick.
> > > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > > >> > > > > > > > > > > > > > > > >>> Regarding
> > deleting/inserting
> > > > it
> > > > > > was
> > > > > > > > >> fixed
> > > > > > > > >> > at
> > > > > > > > >> > > > some
> > > > > > > > >> > > > > > > > point.
> > > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > > >> > > > > > > > > > > > > > > > >>> El mar, 21 nov 2023 a
> las
> > > > 20:22,
> > > > > > > > >> Francisco
> > > > > > > > >> > > > Javier
> > > > > > > > >> > > > > > > > Tirado
> > > > > > > > >> > > > > > > > > > Sarti
> > > > > > > > >> > > > > > > > > > > > > > > > >>> (<ft...@redhat.com>)
> > > > > escribió:
> > > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > Hi Martin,
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > I have a task to
> review
> > > > > > > performance
> > > > > > > > of
> > > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > ProcessInstanceNodeDataEventMerger
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > My idea is to reduce
> the
> > > > > number
> > > > > > of
> > > > > > > > >> delete
> > > > > > > > >> > > > > inserts
> > > > > > > > >> > > > > > > > when
> > > > > > > > >> > > > > > > > > > > > processing
> > > > > > > > >> > > > > > > > > > > > > > > > >>> events
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > and try to do it
> > > > incremental.
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > That should improve
> > > > > performance.
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > PS:
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > I was planning to send
> > an
> > > > > e-mail
> > > > > > > > >> tomorrow
> > > > > > > > >> > > > > > > announcing
> > > > > > > > >> > > > > > > > > > that in
> > > > > > > > >> > > > > > > > > > > > > > case you
> > > > > > > > >> > > > > > > > > > > > > > > > >>> were
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > already working on a
> fix
> > > for
> > > > > > > that. I
> > > > > > > > >> > assume
> > > > > > > > >> > > > you
> > > > > > > > >> > > > > > are
> > > > > > > > >> > > > > > > > not
> > > > > > > > >> > > > > > > > > > and I
> > > > > > > > >> > > > > > > > > > > > > > would
> > > > > > > > >> > > > > > > > > > > > > > > > be
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > sending a PR soon.
> > > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > On Tue, Nov 21, 2023
> at
> > > > > 6:09 PM
> > > > > > > > Martin
> > > > > > > > >> > > Weiler
> > > > > > > > >> > > > > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > wrote:
> > > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > > I looked into the
> new
> > > > > examples
> > > > > > > > using
> > > > > > > > >> > > > > data-index
> > > > > > > > >> > > > > > > > > > persistence
> > > > > > > > >> > > > > > > > > > > > > > addon -
> > > > > > > > >> > > > > > > > > > > > > > > > >>> Neus'
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > > PR#1813 [1] for
> > > serverless
> > > > > and
> > > > > > > > >> Pere's
> > > > > > > > >> > > > branch
> > > > > > > > >> > > > > > [2]
> > > > > > > > >> > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > workflow
> > > > > > > > >> > > > > > > > > > > > > > > > (great
> > > > > > > > >> > > > > > > > > > > > > > > > >>> job
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > > both!) - and they
> work
> > > > > without
> > > > > > > > >> issues
> > > > > > > > >> > > using
> > > > > > > > >> > > > > > > single
> > > > > > > > >> > > > > > > > > > > > requests.
> > > > > > > > >> > > > > > > > > > > > > > > > >>> However, under
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > > some load (I used
> 'ab'
> > > for
> > > > > > > testing
> > > > > > > > >> > with a
> > > > > > > > >> > > > > light
> > > > > > > > >> > > > > > > > > > > > concurrency of
> > > > > > > > >> > > > > > > > > > > > > > 10
> > > > > > > > >> > > > > > > > > > > > > > > > >>> parallel
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > > requests) I ran into
> > the
> > > > > > > following
> > > > > > > > >> > > > problems:
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > > (1) Large number of
> > > > > > > insert/delete
> > > > > > > > >> calls
> > > > > > > > >> > > > (eg.
> > > > > > > > >> > > > > > for
> > > > > > > > >> > > > > > > > > tables
> > > > > > > > >> > > > > > > > > > > > such as
> > > > > > > > >> > > > > > > > > > > > > > > > >>> nodes,
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > > definitions, etc)
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > > (2) Hibernate
> > > > > > > > >> OptimisticLockExceptions
> > > > > > > > >> > /
> > > > > > > > >> > > > > > > > > > > > StaleStateExceptions
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > > (3) DB deadlocks
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > > (4) Error responses,
> > > slow
> > > > > > > response
> > > > > > > > >> > times
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > > The reason I am
> > reaching
> > > > out
> > > > > > > with
> > > > > > > > >> this
> > > > > > > > >> > > > topic
> > > > > > > > >> > > > > > here
> > > > > > > > >> > > > > > > > is
> > > > > > > > >> > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > find
> > > > > > > > >> > > > > > > > > > > > > > out if
> > > > > > > > >> > > > > > > > > > > > > > > > >>> we are
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > > aware of this issue,
> > and
> > > > if
> > > > > > > > someone
> > > > > > > > >> is
> > > > > > > > >> > > > > already
> > > > > > > > >> > > > > > > > > looking
> > > > > > > > >> > > > > > > > > > > > into or
> > > > > > > > >> > > > > > > > > > > > > > > > being
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > > assigned to it?
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > > Thanks,
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > > Martin
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > > [1]
> > > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> >
> > > > > https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > > [2]
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > >
> > > > > > > > >>
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > > To unsubscribe,
> > e-mail:
> > > > > > > > >> > > > > > > > > dev-unsubscribe@kie.apache.org
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > > For additional
> > commands,
> > > > > > e-mail:
> > > > > > > > >> > > > > > > > > > dev-help@kie.apache.org
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > >> > > > > > > > > > > > > > > > >>> To unsubscribe, e-mail:
> > > > > > > > >> > > > > > > dev-unsubscribe@kie.apache.org
> > > > > > > > >> > > > > > > > > > > > > > > > >>> For additional commands,
> > > > e-mail:
> > > > > > > > >> > > > > > > > dev-help@kie.apache.org
> > > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > >
> > > > > > > > >>
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > >> > > > > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > > > > > > >> > > > > > dev-unsubscribe@kie.apache.org
> > > > > > > > >> > > > > > > > > > > > > > > > > For additional commands,
> > > e-mail:
> > > > > > > > >> > > > > > > dev-help@kie.apache.org
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > >> > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > >
> >
>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Francisco Javier Tirado Sarti <ft...@redhat.com>]

Hi Enrique,
In the case of data index I think the data to be purged is finished process
instances (I do not think we should remove process instance that has been
alive for ages, even if it is very likely they are not going to be ever
completed)
Once you delete those process instances, you also delete the associated
user tasks and jobs.
Therefore the problem is relatively simple, to be able to configure how
much time a completed process instance should remain in the data index
database. We can take a simple approach: a property with a min duration
that cannot be changed once data index is started; a slightly complex one:
the same property but watching it to react for changes; or the full suite:
an admin API to be able to change the policy at any moment.
I think this discussion is worth having.

On Wed, Feb 21, 2024 at 6:15 AM Enrique Gonzalez Martinez <
egonzalez@apache.org> wrote:

> Hi Martin, the main problem regarding the purge is because it is still
> unclear the policy for what tech to use and the future components.
>
> Recently we had a discussion about proposing graphql for this sort of admin
> tasks. So far for subsystems we have been using rest endpoints (like update
> timers or modify human task or change processes). There is one exception
> which is the gateway that is pure graphql and somehow uses graphql for
> everything making complex operations under the hood.
>
> This has somehow frozen the purge for data audit for a bit and the proposal
> was to use rest endpoints to do the clean up in the component and offer the
> graphql counterpart in the gateway promoting it to a first class citizen
> component instead of having it embedded in the data index.
>
> I would suggest to come up at least with a policy first regarding the
> convention every component should address this.
>
>
> El mar, 20 feb 2024, 23:39, Martin Weiler <mw...@ibm.com.invalid>
> escribió:
>
> > IMO, it is good to have this discussion around data sanity now instead of
> > putting it off until later when data has already accumulated in
> production
> > environments.
> >
> > Based on the input here, we are dealing with three types of data:
> > 1. Runtime data - active instances only, engine cleans up the data
> > automatically at process instance end
> > 2. Historic log data - data created by data-audit intended for long term
> > storage
> > 3. Data-index data - somehow this data falls in between the two
> > aforementioned categories, with the idea of the data being "recent", but
> > not restricted to active instances only
> >
> > We'd need purge strategies for both #2 and #3 (perhaps different ones, or
> > with different config settings) in order to prevent unlimited data
> growth.
> >
> > ________________________________________
> > From: Enrique Gonzalez Martinez <eg...@apache.org>
> > Sent: Monday, February 19, 2024 7:11 AM
> > To: dev@kie.apache.org
> > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with data-index
> > persistence addon
> >
> > Hi Francisco,
> > To give you more context about this.
> >
> > STP is a concept, a process with certain constraints: no persistence and
> > returning the outcome in the call (sync execution with no idle states).
> It
> > was a requirement from a user in the past. One of the requirements was
> > leaving no trail. In v7 was easy because you could disable the audit in
> > that case. Actually we have the same way to do what we did in v7 in here
> as
> > you can add/remove index just removing deps.
> >
> > We have the same outcome with different approaches and STP is already
> > delivered.
> >
> > El lun, 19 feb 2024 a las 14:46, Francisco Javier Tirado Sarti (<
> > ftirados@redhat.com>) escribió:
> >
> > > Regarding STP (which is not a concept that we have in the code. I mean
> > STP
> > > are processes as nonSTP are), I guess, as all processes, they were kept
> > in
> > > DataIndex once completed because users wanted (and still wants) to
> check
> > > the result once the call had been performed. If we want to leave no
> trace
> > > of them in DataIndex for some reason, we will need to make it a
> > > Runtimes concept so DataIndex can handle them in a different way.
> > >
> > > On Mon, Feb 19, 2024 at 2:27 PM Enrique Gonzalez Martinez <
> > > egonzalez@apache.org> wrote:
> > >
> > > > Alex:
> > > > Right now the data index is working in the same way as it did in v7
> > with
> > > > the emitters. The only difference between two impl is that in here
> the
> > > > storage is pgsql instead elastic search.  You are right regarding is
> a
> > > > snapshot of the last state of the process but we did never define how
> > > long
> > > > would be alive that dats Honestly i am happy right now with the way
> it
> > > > works. The clean up mechanism is still tbd because we still need to
> > > discuss
> > > > other stuff first.
> > > >
> > > >
> > > > Regarding stp is to leave no trail because u can get the outcome
> > directly
> > > > from the call. It was defined like that in v7. So there is no use for
> > the
> > > > index or the audit.
> > > >
> > > > El lun, 19 feb 2024, 14:13, Francisco Javier Tirado Sarti <
> > > > ftirados@redhat.com> escribió:
> > > >
> > > > > Hi Alex,
> > > > > There has been some confusion about the purpose of DataIndex. To be
> > > > honest
> > > > > I believe they were already sorted out, but your e-mail makes me
> > think
> > > > that
> > > > > is not the case ;). I let Kris to clarify that with you. My view is
> > > that
> > > > > data-index is a way to query recently closed and active processes
> > (the
> > > > key
> > > > > here is the definition of recently, which in my opinion should be
> > > > > configurable)
> > > > > But, besides that discussion and being pragmatic, keeping finishing
> > > > process
> > > > > instances "for a while" in DataIndex was the only way for users to
> > > query
> > > > > the result of straight through processes. That's a function that
> > cannot
> > > > be
> > > > > removed right now
> > > > >
> > > > > On Mon, Feb 19, 2024 at 1:33 PM Alex Porcelli <porcelli@apache.org
> >
> > > > wrote:
> > > > >
> > > > > > if data index was supposed to provide snapshot view of the
> process
> > > > > > instance… why do we keep it after the process instance is
> finished?
> > > > > >
> > > > > >
> > > > > > On Mon, Feb 19, 2024 at 7:12 AM Francisco Javier Tirado Sarti <
> > > > > > ftirados@redhat.com> wrote:
> > > > > >
> > > > > > > Hi Martin.
> > > > > > > After taking a deeper look at this, I realize that the
> behaviour
> > is
> > > > the
> > > > > > > expected one.
> > > > > > > Runtimes DB does not track the completed process instance
> (that's
> > > > what
> > > > > > the
> > > > > > > JDBCProcessInstances warn is telling us), but DataIndex, as
> > > expected,
> > > > > is
> > > > > > > tracking it in processes and nodes table. And yes it will grow
> > over
> > > > > time.
> > > > > > > What we need is some configurable purge mechanism for
> DataIndex,
> > so
> > > > it
> > > > > > > eventually removes older completed process instances.
> > > > > > >
> > > > > > > On Tue, Feb 13, 2024 at 12:59 PM Francisco Javier Tirado Sarti
> <
> > > > > > > ftirados@redhat.com> wrote:
> > > > > > >
> > > > > > > > Hi Martin,
> > > > > > > > Good catch!. Looks like the skipping performed for process
> > > > instances
> > > > > is
> > > > > > > > not applied to node instances. Something we definitely need
> to
> > > > review
> > > > > > on
> > > > > > > > the runtimes side.
> > > > > > > >
> > > > > > > > On Mon, Feb 12, 2024 at 11:59 PM Martin Weiler
> > > > > <mweiler@ibm.com.invalid
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> On a somewhat related note, testing a simple workflow (start
> > ->
> > > > > script
> > > > > > > >> node -> end), I see the following messages in the logs:
> > > > > > > >> 2024-02-12 22:49:50,493 28758dde544c WARN
> > > > > > > >> [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> > > > > > > >> (executor-thread-3) Skipping create of process instance id:
> > > > > > > >> 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2
> > > > > > > >>
> > > > > > > >> So far, so good. And I'd expect to see no trace of this
> > process
> > > in
> > > > > the
> > > > > > > >> database if I don't have data audit enabled.
> > > > > > > >>
> > > > > > > >> However, the 'processes' table contains a row with state=2,
> > with
> > > > > > related
> > > > > > > >> entries in the 'nodes' table. In a load test, I see these
> > tables
> > > > > grow
> > > > > > > >> significantly over time. Am I missing something to have
> these
> > > > > entries
> > > > > > > >> cleaned up automatically?
> > > > > > > >>
> > > > > > > >> ________________________________________
> > > > > > > >> From: Martin Weiler <mw...@ibm.com.INVALID>
> > > > > > > >> Sent: Monday, February 12, 2024 3:40 PM
> > > > > > > >> To: dev@kie.apache.org
> > > > > > > >> Subject: [EXTERNAL] RE: [DISCUSSION] Performance issues with
> > > > > > data-index
> > > > > > > >> persistence addon
> > > > > > > >>
> > > > > > > >> Thanks everyone for your input. Based on this discussion, I
> > > opened
> > > > > the
> > > > > > > >> following PR:
> > > > > > > >>
> https://github.com/apache/incubator-kie-kogito-apps/pull/1985
> > > > > > > >>
> > > > > > > >> With this change, the performance seems to be stable over
> > time:
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
> > > > > > > >>
> > > > > > > >> Martin
> > > > > > > >>
> > > > > > > >> ________________________________________
> > > > > > > >> From: Gonzalo Muñoz <gm...@apache.org>
> > > > > > > >> Sent: Friday, February 9, 2024 9:42 AM
> > > > > > > >> To: dev@kie.apache.org
> > > > > > > >> Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > > > > > data-index
> > > > > > > >> persistence addon
> > > > > > > >>
> > > > > > > >> Great work Francisco,
> > > > > > > >> Martin, take a look at this link with some related tips (in
> > case
> > > > you
> > > > > > > find
> > > > > > > >> it useful):
> > > > > > > >>
> > https://www.cybertec-postgresql.com/en/index-your-foreign-key/
> > > > > > > >>
> > > > > > > >> El vie, 9 feb 2024 a las 17:20, Francisco Javier Tirado
> Sarti
> > (<
> > > > > > > >> ftirados@redhat.com>) escribió:
> > > > > > > >>
> > > > > > > >> > For the moment being, we will keep JPA till we exhaust all
> > > > > > > >> possibilities,
> > > > > > > >> > let's call switching from jpa to jdbc our hidden plan B ;)
> > > > > > > >> > I already told Martin, but in order everyone to know, just
> > > after
> > > > > > > writing
> > > > > > > >> > the previous email, I thought "what if Postgres is not
> > > > > automatically
> > > > > > > >> > indexing foreign keys like mysql?" and, eureka
> > > > > > > >> > Postgres doc
> > > > > > > >> >
> > https://www.postgresql.org/docs/current/ddl-constraints.html
> > > > > > > >> > Mysql doc
> > > > > > > >> >
> > > > >
> https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> > > > > > > >> > These are the relevant excerpt
> > > > > > > >> >
> > > > > > > >> > *Postgresql*
> > > > > > > >> > *A foreign key must reference columns that either are a
> > > primary
> > > > > key
> > > > > > or
> > > > > > > >> form
> > > > > > > >> > a unique constraint, or are columns from a non-partial
> > unique
> > > > > index.
> > > > > > > >> This
> > > > > > > >> > means that the referenced columns always have an index to
> > > allow
> > > > > > > >> efficient
> > > > > > > >> > lookups on whether a referencing row has a match. Since a
> > > DELETE
> > > > > of
> > > > > > a
> > > > > > > >> row
> > > > > > > >> > from the referenced table or an UPDATE of a referenced
> > column
> > > > will
> > > > > > > >> require
> > > > > > > >> > a scan of the referencing table for rows matching the old
> > > value,
> > > > > it
> > > > > > is
> > > > > > > >> > often a good idea to index the referencing columns too.
> > > Because
> > > > > this
> > > > > > > is
> > > > > > > >> not
> > > > > > > >> > always needed, and there are many choices available on how
> > to
> > > > > index,
> > > > > > > the
> > > > > > > >> > declaration of a foreign key constraint does not
> > automatically
> > > > > > create
> > > > > > > an
> > > > > > > >> > index on the referencing columns.*
> > > > > > > >> > *Mysql*
> > > > > > > >> > *MySQL requires that foreign key columns be indexed; if
> you
> > > > > create a
> > > > > > > >> table
> > > > > > > >> > with a foreign key constraint but no index on a given
> > column,
> > > an
> > > > > > index
> > > > > > > >> is
> > > > > > > >> > created. *
> > > > > > > >> >
> > > > > > > >> > So I asked Martin to especially create an index for
> > > > > > > process_instance_id
> > > > > > > >> > column on nodes table
> > > > > > > >> > I think that will fix the problem detected on the thread
> > dump.
> > > > > > > >> > The simpler process test to verify queries are fine still
> > > > stands,
> > > > > > > >> though ;)
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <
> > > > tzimanyi@apache.org
> > > > > >
> > > > > > > >> wrote:
> > > > > > > >> >
> > > > > > > >> > > I always preferred pure JDBC over Hibernate myself, just
> > for
> > > > the
> > > > > > > sake
> > > > > > > >> of
> > > > > > > >> > > control of what is happening :) So I would not -1 that
> > > myself.
> > > > > > > >> > >
> > > > > > > >> > > Tibor
> > > > > > > >> > >
> > > > > > > >> > > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado Sarti <
> > > > > > > >> > > ftirados@redhat.com>
> > > > > > > >> > > napísal(a):
> > > > > > > >> > >
> > > > > > > >> > > > Hi,
> > > > > > > >> > > > Usually I do not want to talk about work in progress
> > > because
> > > > > > > >> > preliminary
> > > > > > > >> > > > conclusions are pretty volatile but, well, there are a
> > > > couple
> > > > > of
> > > > > > > >> things
> > > > > > > >> > > > that can be concluded from the really valuable
> > information
> > > > > that
> > > > > > > >> Martin
> > > > > > > >> > > > provided.
> > > > > > > >> > > > 1) In order to be able to determine if the number of
> > > > > statements
> > > > > > is
> > > > > > > >> > larger
> > > > > > > >> > > > than expected, I asked Martin to test with a simpler
> > > process
> > > > > > > >> > definition.
> > > > > > > >> > > > One with just three nodes: start, script and end. The
> > > script
> > > > > one
> > > > > > > >> should
> > > > > > > >> > > > change just one variable. This way we can analyze if
> the
> > > > > number
> > > > > > of
> > > > > > > >> > > queries
> > > > > > > >> > > > is the expected one. From the single log (audit was
> > > > activated
> > > > > > > them)
> > > > > > > >> my
> > > > > > > >> > > > conclusion is that the number of insert/updates over
> > > > processes
> > > > > > and
> > > > > > > >> > nodes
> > > > > > > >> > > > (there a lot over task, that I will prefer to skip for
> > > now,
> > > > > baby
> > > > > > > >> steps)
> > > > > > > >> > > is
> > > > > > > >> > > > the expected one.
> > > > > > > >> > > > 2) Analysing the thread dump, we see around 15 threads
> > > > > executing
> > > > > > > >> this
> > > > > > > >> > > line
> > > > > > > >> > > > at
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > > > > > >> > > > so its pretty clear the code to be optimized ;). I'm
> > > > > evaluating
> > > > > > > >> > > > possibilities within JPA/Hibernate, but I'm starting
> to
> > > > think
> > > > > > that
> > > > > > > >> it
> > > > > > > >> > > might
> > > > > > > >> > > > be better to switch to JDBC and skip hibernate. Our
> > lives
> > > > will
> > > > > > be
> > > > > > > >> > > simpler,
> > > > > > > >> > > > especially with a schema relatively simple like ours
> > (that
> > > > > will
> > > > > > be
> > > > > > > >> my
> > > > > > > >> > > > recommendation if I was an external consultant)
> > > > > > > >> > > >
> > > > > > > >> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <
> > > > > > tzimanyi@apache.org
> > > > > > > >
> > > > > > > >> > > wrote:
> > > > > > > >> > > >
> > > > > > > >> > > > > Hi,
> > > > > > > >> > > > >
> > > > > > > >> > > > > this will be a bit off-topic. However as far as
> > > > > performance, I
> > > > > > > >> think
> > > > > > > >> > we
> > > > > > > >> > > > > should think about that we have string primary keys
> > > > (IDs). I
> > > > > > > would
> > > > > > > >> > > expect
> > > > > > > >> > > > > the database systems are much better with indexing
> > > numeric
> > > > > > keys
> > > > > > > >> than
> > > > > > > >> > > > > strings. I remember from the past, when I was
> working
> > > with
> > > > > > DBs,
> > > > > > > >> that
> > > > > > > >> > > > using
> > > > > > > >> > > > > strings as keys or indexes was a discouraged
> practice.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Best regards,
> > > > > > > >> > > > > Tibor
> > > > > > > >> > > > >
> > > > > > > >> > > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler
> > > > > > <mweiler@ibm.com.invalid
> > > > > > > >
> > > > > > > >> > > > > napísal(a):
> > > > > > > >> > > > >
> > > > > > > >> > > > > > I changed the test to use MongoDB [1] and I don't
> > see
> > > a
> > > > > > > >> performance
> > > > > > > >> > > > > > degradation with this setup [2].
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Please keep us posted of your findings. Thanks!
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Martin
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > [1]
> > > > > > > >> > > > >
> > > > > > > >> >
> > > > > > >
> > > >
> https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > > > > >> > > > > > [2]
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > ________________________________________
> > > > > > > >> > > > > > From: Francisco Javier Tirado Sarti <
> > > > ftirados@redhat.com>
> > > > > > > >> > > > > > Sent: Wednesday, February 7, 2024 11:40 AM
> > > > > > > >> > > > > > To: dev@kie.apache.org
> > > > > > > >> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance
> > > issues
> > > > > with
> > > > > > > >> > > data-index
> > > > > > > >> > > > > > persistence addon
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > yes, it can be index degradation because of size,
> > but
> > > I
> > > > > > > believe
> > > > > > > >> (I
> > > > > > > >> > > > might
> > > > > > > >> > > > > be
> > > > > > > >> > > > > > wrong) the db is too small (yet) for that.
> > > > > > > >> > > > > > But, eventually, Postgres, when the DB is huge
> > enough,
> > > > > > > >> unavoidably
> > > > > > > >> > > will
> > > > > > > >> > > > > > behave like the graphic that Martin sent.
> > > > > > > >> > > > > > Since I believe we are not huge enough (yet), lets
> > > rule
> > > > > out
> > > > > > > >> another
> > > > > > > >> > > > issue
> > > > > > > >> > > > > > by analysing the sql logs (I requested those to
> > Martin
> > > > > > offline
> > > > > > > >> and
> > > > > > > >> > he
> > > > > > > >> > > > is
> > > > > > > >> > > > > > going to kindly collect them).
> > > > > > > >> > > > > > Also Im curious to know if Mongo behave in the
> same
> > > way.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez
> > > > Martinez <
> > > > > > > >> > > > > > egonzalez@apache.org> wrote:
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > Hi Francisco,
> > > > > > > >> > > > > > > I would highly recommend to check indexes and
> how
> > > the
> > > > > > > updates
> > > > > > > >> > work
> > > > > > > >> > > in
> > > > > > > >> > > > > > data
> > > > > > > >> > > > > > > index to avoid full scan table and lock the full
> > > > table.
> > > > > > Some
> > > > > > > >> db
> > > > > > > >> > are
> > > > > > > >> > > > > very
> > > > > > > >> > > > > > > sensitive to that.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > El mié, 7 feb 2024, 18:41, Francisco Javier
> Tirado
> > > > > Sarti <
> > > > > > > >> > > > > > > ftirados@redhat.com> escribió:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > > Hi Martin,
> > > > > > > >> > > > > > > > While I analyze the data, let me ask you if it
> > is
> > > > > > possible
> > > > > > > >> to
> > > > > > > >> > > > perform
> > > > > > > >> > > > > > > > another check (similar in a way to disabling
> > > > > data-index
> > > > > > > like
> > > > > > > >> > you
> > > > > > > >> > > > do)
> > > > > > > >> > > > > > Can
> > > > > > > >> > > > > > > > you switch to MongoDB persistence and check if
> > the
> > > > > same
> > > > > > > >> > > degradation
> > > > > > > >> > > > > > that
> > > > > > > >> > > > > > > is
> > > > > > > >> > > > > > > > there for postgres remains?
> > > > > > > >> > > > > > > > I do not know if this is feasible but will
> > > certainly
> > > > > > > >> indicate
> > > > > > > >> > the
> > > > > > > >> > > > > > problem
> > > > > > > >> > > > > > > > is on the postgres storage layer and I do not
> > > have a
> > > > > > clear
> > > > > > > >> > > > prediction
> > > > > > > >> > > > > > of
> > > > > > > >> > > > > > > > what we will see when doing this switch.
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler
> > > > > > > >> > > > <mweiler@ibm.com.invalid
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > > wrote:
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > > Hi Francisco,
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > thanks for your work on this important
> topic!
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > I would like to share some test results
> here,
> > > > which
> > > > > > > might
> > > > > > > >> > help
> > > > > > > >> > > to
> > > > > > > >> > > > > > > improve
> > > > > > > >> > > > > > > > > the codebase even further. I am using the
> > jmeter
> > > > > based
> > > > > > > >> test
> > > > > > > >> > > case
> > > > > > > >> > > > > from
> > > > > > > >> > > > > > > > Pere
> > > > > > > >> > > > > > > > > and Enrique (thanks guys!) [1] which uses a
> > load
> > > > of
> > > > > 30
> > > > > > > >> > threads
> > > > > > > >> > > to
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > 1) start a new process instance (POST)
> > > > > > > >> > > > > > > > > 2) retrieve tasks for a user (GET)
> > > > > > > >> > > > > > > > > 3) fetches task details (GET)
> > > > > > > >> > > > > > > > > 4) complete a task (POST)
> > > > > > > >> > > > > > > > > 5) execute a query on data-audit
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > With this test setup, I noticed that the
> > > > performance
> > > > > > for
> > > > > > > >> the
> > > > > > > >> > > POST
> > > > > > > >> > > > > > > > > requests, in particular the one to start a
> new
> > > > > process
> > > > > > > >> > > instance,
> > > > > > > >> > > > > > > degrades
> > > > > > > >> > > > > > > > > over time - see graph [2]. If I run the same
> > > test
> > > > > > > without
> > > > > > > >> > > > > data-index,
> > > > > > > >> > > > > > > > then
> > > > > > > >> > > > > > > > > there is no such performance degradation
> [3].
> > > You
> > > > > can
> > > > > > > >> find a
> > > > > > > >> > > > thread
> > > > > > > >> > > > > > > dump
> > > > > > > >> > > > > > > > > captured a few minutes into the first test
> > here
> > > > [4]
> > > > > > that
> > > > > > > >> > might
> > > > > > > >> > > > help
> > > > > > > >> > > > > > to
> > > > > > > >> > > > > > > > see
> > > > > > > >> > > > > > > > > some of the contention points.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > I'd appreciate if you could take a look and
> > see
> > > if
> > > > > > there
> > > > > > > >> is
> > > > > > > >> > > > > something
> > > > > > > >> > > > > > > > that
> > > > > > > >> > > > > > > > > can be further improved based on your
> previous
> > > > work.
> > > > > > If
> > > > > > > >> you
> > > > > > > >> > > need
> > > > > > > >> > > > > any
> > > > > > > >> > > > > > > > > additional data, let me know, but otherwise
> it
> > > is
> > > > > > > >> > > straightforward
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > > run
> > > > > > > >> > > > > > > > > the jmeter test as well.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Thanks,
> > > > > > > >> > > > > > > > > Martin
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > [1]
> > > > > > > >> https://github.com/pefernan/job-service-refactor-test/
> > > > > > > >> > > > > > > > > [2]
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > > > >> > > > > > > > > [3]
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > > > >> > > > > > > > > [4]
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > ________________________________________
> > > > > > > >> > > > > > > > > From: Francisco Javier Tirado Sarti <
> > > > > > > ftirados@redhat.com>
> > > > > > > >> > > > > > > > > Sent: Wednesday, January 17, 2024 9:13 AM
> > > > > > > >> > > > > > > > > To: dev@kie.apache.org
> > > > > > > >> > > > > > > > > Cc: Pere Fernandez Perez
> > > > > > > >> > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION]
> > Performance
> > > > > > issues
> > > > > > > >> with
> > > > > > > >> > > > > > data-index
> > > > > > > >> > > > > > > > > persistence addon
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Hi Alex,
> > > > > > > >> > > > > > > > > I did not take times (which depends on a
> > number
> > > of
> > > > > > > >> variables
> > > > > > > >> > > that
> > > > > > > >> > > > > > > > > drastically change between environments),
> but
> > > > verify
> > > > > > > that
> > > > > > > >> the
> > > > > > > >> > > > > number
> > > > > > > >> > > > > > of
> > > > > > > >> > > > > > > > > updates has been reduced drastically without
> > > > losing
> > > > > > > >> > > > functionality,
> > > > > > > >> > > > > > > which
> > > > > > > >> > > > > > > > is
> > > > > > > >> > > > > > > > > objectively a good thing. If before the
> > change,
> > > > for
> > > > > > > every
> > > > > > > >> > node
> > > > > > > >> > > > > > > executed,
> > > > > > > >> > > > > > > > we
> > > > > > > >> > > > > > > > > have an update for every node previously
> > > executed,
> > > > > so
> > > > > > > if a
> > > > > > > >> > > > process
> > > > > > > >> > > > > > have
> > > > > > > >> > > > > > > > 50
> > > > > > > >> > > > > > > > > nodes to execute, we were performing nearly
> > > > 50*51/2
> > > > > > > >> updates,
> > > > > > > >> > > > which
> > > > > > > >> > > > > > > gives
> > > > > > > >> > > > > > > > us
> > > > > > > >> > > > > > > > > a total of  1275 updates, now we have just
> one
> > > for
> > > > > > every
> > > > > > > >> node
> > > > > > > >> > > > being
> > > > > > > >> > > > > > > > > executed, implying a total of 50 updates.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex
> Porcelli
> > <
> > > > > > > >> > > alex@porcelli.me>
> > > > > > > >> > > > > > > wrote:
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > > Francisco,
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > I noticed that your PR has been merged,
> but
> > I
> > > > was
> > > > > > > >> expecting
> > > > > > > >> > > (at
> > > > > > > >> > > > > > least
> > > > > > > >> > > > > > > > > > was my understanding from this thread)
> that
> > > > before
> > > > > > > >> merging
> > > > > > > >> > > some
> > > > > > > >> > > > > > > > > > benchmark data would be shared in advance
> -
> > to
> > > > > > assess
> > > > > > > >> the
> > > > > > > >> > > > > > > cost/benefit
> > > > > > > >> > > > > > > > > > of such a decent size change.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Do you have any information to share?
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM Francisco
> > > Javier
> > > > > > > Tirado
> > > > > > > >> > Sarti
> > > > > > > >> > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Yes, as intended, now we have one select
> > and
> > > > one
> > > > > > > >> > > > insert/update
> > > > > > > >> > > > > > per
> > > > > > > >> > > > > > > > node
> > > > > > > >> > > > > > > > > > > event.
> > > > > > > >> > > > > > > > > > > I moved the PR as ready for review and
> > give
> > > > > @Pere
> > > > > > > >> > Fernandez
> > > > > > > >> > > > > Perez
> > > > > > > >> > > > > > > > > > > <pe...@redhat.com> permission to the
> > > > branch
> > > > > so
> > > > > > > he
> > > > > > > >> can
> > > > > > > >> > > > edit
> > > > > > > >> > > > > it
> > > > > > > >> > > > > > > in
> > > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > > > > > > next two weeks (Ill be on PTO)  if
> > desired,
> > > > > before
> > > > > > > >> > merging.
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex
> > > Porcelli
> > > > <
> > > > > > > >> > > > > alex@porcelli.me>
> > > > > > > >> > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > Cool, thank you Francisco!
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > Did you manage to get some preliminary
> > > data
> > > > > > about
> > > > > > > >> > > > > improvements?
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM
> > Francisco
> > > > > > Javier
> > > > > > > >> > Tirado
> > > > > > > >> > > > > Sarti
> > > > > > > >> > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > Yes, after some delay because of
> > > quarkus 3
> > > > > > > >> migration.
> > > > > > > >> > > Im
> > > > > > > >> > > > > > > refining
> > > > > > > >> > > > > > > > > > this
> > > > > > > >> > > > > > > > > > > > > draft PR
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > >
> > > > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex
> > > > > Porcelli
> > > > > > <
> > > > > > > >> > > > > > > alex@porcelli.me>
> > > > > > > >> > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > Any update or new findings on this
> > > > topic?
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM
> > > > Francisco
> > > > > > > Javier
> > > > > > > >> > > Tirado
> > > > > > > >> > > > > > Sarti
> > > > > > > >> > > > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > Hi Alex,
> > > > > > > >> > > > > > > > > > > > > > > After considering different
> > options
> > > to
> > > > > > > improve
> > > > > > > >> > > > > > performance,
> > > > > > > >> > > > > > > > we
> > > > > > > >> > > > > > > > > > feel
> > > > > > > >> > > > > > > > > > > > that
> > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > >> > > > > > > > > > > > > > > is time to "partially" move away
> > > from
> > > > > the
> > > > > > > >> current
> > > > > > > >> > > Map
> > > > > > > >> > > > > > style
> > > > > > > >> > > > > > > > > > > > interface (
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > > >> > > > > > > > > > > > > > )
> > > > > > > >> > > > > > > > > > > > > > > which was shared with Trusty, to
> > one
> > > > > more
> > > > > > > >> > suitable
> > > > > > > >> > > > for
> > > > > > > >> > > > > > > usage
> > > > > > > >> > > > > > > > > > with a
> > > > > > > >> > > > > > > > > > > > > > > relational DB like postgresql
> (but
> > > > still
> > > > > > > >> > compatible
> > > > > > > >> > > > > with
> > > > > > > >> > > > > > > big
> > > > > > > >> > > > > > > > > > table
> > > > > > > >> > > > > > > > > > > > dbs).
> > > > > > > >> > > > > > > > > > > > > > > The idea will be to replace
> > generic
> > > > > > Storage
> > > > > > > >> > > interface
> > > > > > > >> > > > > by
> > > > > > > >> > > > > > > four
> > > > > > > >> > > > > > > > > > > > specific
> > > > > > > >> > > > > > > > > > > > > > > interfaces (which will inherit
> > from
> > > a
> > > > > > common
> > > > > > > >> one
> > > > > > > >> > > that
> > > > > > > >> > > > > > keeps
> > > > > > > >> > > > > > > > the
> > > > > > > >> > > > > > > > > > query
> > > > > > > >> > > > > > > > > > > > > > part
> > > > > > > >> > > > > > > > > > > > > > > at is it. with get and query
> > > methods),
> > > > > > that
> > > > > > > >> will
> > > > > > > >> > > > > include
> > > > > > > >> > > > > > > the
> > > > > > > >> > > > > > > > > > required
> > > > > > > >> > > > > > > > > > > > > > > modification operations for the
> > four
> > > > > > > DataIndex
> > > > > > > >> > > > > "domains":
> > > > > > > >> > > > > > > > > > > > > > processinstance,
> > > > > > > >> > > > > > > > > > > > > > > usertask, processdefinitions and
> > > jobs.
> > > > > > Those
> > > > > > > >> > > > interfaces
> > > > > > > >> > > > > > > will
> > > > > > > >> > > > > > > > > > define
> > > > > > > >> > > > > > > > > > > > > > methods
> > > > > > > >> > > > > > > > > > > > > > > like addNode, addVariable,
> > > updateTask,
> > > > > > > >> > > > > addAttachment.....
> > > > > > > >> > > > > > > > that
> > > > > > > >> > > > > > > > > > will
> > > > > > > >> > > > > > > > > > > > allow
> > > > > > > >> > > > > > > > > > > > > > > the persistent layer
> > implementation
> > > > to
> > > > > > just
> > > > > > > >> > update
> > > > > > > >> > > > the
> > > > > > > >> > > > > > > > needed
> > > > > > > >> > > > > > > > > > info
> > > > > > > >> > > > > > > > > > > > in
> > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > DB  (for example, for addNode in
> > > > > Postgres,
> > > > > > > >> just
> > > > > > > >> > > > insert
> > > > > > > >> > > > > a
> > > > > > > >> > > > > > > row
> > > > > > > >> > > > > > > > > into
> > > > > > > >> > > > > > > > > > > > nodes
> > > > > > > >> > > > > > > > > > > > > > > table, for addNode in Mongo,
> > > basically
> > > > > the
> > > > > > > >> same
> > > > > > > >> > > > atomic
> > > > > > > >> > > > > > > upsert
> > > > > > > >> > > > > > > > > > > > operation
> > > > > > > >> > > > > > > > > > > > > > > that is currently done).
> > Therefore,
> > > we
> > > > > > > >> increase
> > > > > > > >> > > > > > performance
> > > > > > > >> > > > > > > > for
> > > > > > > >> > > > > > > > > > > > Postgres
> > > > > > > >> > > > > > > > > > > > > > > and keep the current one for
> > Mongo.
> > > > The
> > > > > > > >> current
> > > > > > > >> > DB
> > > > > > > >> > > > > > schemas
> > > > > > > >> > > > > > > > > won't
> > > > > > > >> > > > > > > > > > be
> > > > > > > >> > > > > > > > > > > > > > > touched.
> > > > > > > >> > > > > > > > > > > > > > > Since the code change is large,
> I
> > do
> > > > not
> > > > > > > think
> > > > > > > >> > I'll
> > > > > > > >> > > > be
> > > > > > > >> > > > > > able
> > > > > > > >> > > > > > > > to
> > > > > > > >> > > > > > > > > > have
> > > > > > > >> > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > PR
> > > > > > > >> > > > > > > > > > > > > > > ready till next week.
> > > > > > > >> > > > > > > > > > > > > > > But before starting, please let
> me
> > > > know
> > > > > if
> > > > > > > >> that
> > > > > > > >> > > > > approach
> > > > > > > >> > > > > > is
> > > > > > > >> > > > > > > > > fine
> > > > > > > >> > > > > > > > > > for
> > > > > > > >> > > > > > > > > > > > you.
> > > > > > > >> > > > > > > > > > > > > > > Best regards.
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 6:55 PM
> > Alex
> > > > > > > Porcelli
> > > > > > > >> <
> > > > > > > >> > > > > > > > > alex@porcelli.me>
> > > > > > > >> > > > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > Thank you Francisco to getting
> > > > deeper
> > > > > on
> > > > > > > >> this…
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > Looking forward to see the
> > results
> > > > of
> > > > > > your
> > > > > > > >> > > > suggested
> > > > > > > >> > > > > > > > > > improvements.
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> 9:40 AM
> > > > > > Francisco
> > > > > > > >> > Javier
> > > > > > > >> > > > > Tirado
> > > > > > > >> > > > > > > > > Sarti <
> > > > > > > >> > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > I forgot to attach the
> queries
> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> > 3:04 PM
> > > > > > > Francisco
> > > > > > > >> > > Javier
> > > > > > > >> > > > > > Tirado
> > > > > > > >> > > > > > > > > > Sarti <
> > > > > > > >> > > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > >> Hi,
> > > > > > > >> > > > > > > > > > > > > > > > >> A brief update on this
> topic.
> > > > > > > >> > > > > > > > > > > > > > > > >> After doing a simple test
> > with
> > > > > > example
> > > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > >> > > > > > > > > > > > > > > > ,
> > > > > > > >> > > > > > > > > > > > > > > > >> the number of updates over
> > > Nodes
> > > > > > table
> > > > > > > is
> > > > > > > >> > n*n,
> > > > > > > >> > > > so
> > > > > > > >> > > > > we
> > > > > > > >> > > > > > > > > manage
> > > > > > > >> > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > obtain a
> > > > > > > >> > > > > > > > > > > > > > > > >> perfect quadratic
> performance
> > > > > > > >> degradation.
> > > > > > > >> > The
> > > > > > > >> > > > > > problem
> > > > > > > >> > > > > > > > is
> > > > > > > >> > > > > > > > > > worse
> > > > > > > >> > > > > > > > > > > > in
> > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > case
> > > > > > > >> > > > > > > > > > > > > > > > >> of Serverless Workflow than
> > in
> > > > BPMN
> > > > > > > >> because
> > > > > > > >> > we
> > > > > > > >> > > > the
> > > > > > > >> > > > > > > > number
> > > > > > > >> > > > > > > > > of
> > > > > > > >> > > > > > > > > > > > nodes
> > > > > > > >> > > > > > > > > > > > > > is
> > > > > > > >> > > > > > > > > > > > > > > > >> greater than the number of
> > > > states.
> > > > > In
> > > > > > > >> that
> > > > > > > >> > > > > example N
> > > > > > > >> > > > > > > is
> > > > > > > >> > > > > > > > > 16,
> > > > > > > >> > > > > > > > > > but
> > > > > > > >> > > > > > > > > > > > for
> > > > > > > >> > > > > > > > > > > > > > a
> > > > > > > >> > > > > > > > > > > > > > > > more
> > > > > > > >> > > > > > > > > > > > > > > > >> complex workflow it would
> be
> > > > > > certainly
> > > > > > > >> > large.
> > > > > > > >> > > > > > > > > > > > > > > > >> I think that this is more
> > > related
> > > > > to
> > > > > > > how
> > > > > > > >> we
> > > > > > > >> > > are
> > > > > > > >> > > > > > > handling
> > > > > > > >> > > > > > > > > > JPA in
> > > > > > > >> > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > code,
> > > > > > > >> > > > > > > > > > > > > > > > >> in particular the mapping
> > from
> > > > > model
> > > > > > to
> > > > > > > >> > entity
> > > > > > > >> > > > > > > > (basically
> > > > > > > >> > > > > > > > > > JPA is
> > > > > > > >> > > > > > > > > > > > > > blind
> > > > > > > >> > > > > > > > > > > > > > > > and
> > > > > > > >> > > > > > > > > > > > > > > > >> has to update all nodes for
> > > every
> > > > > > write
> > > > > > > >> > > because
> > > > > > > >> > > > it
> > > > > > > >> > > > > > > > > believes
> > > > > > > >> > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > node has
> > > > > > > >> > > > > > > > > > > > > > > > >> been updated, although it
> is
> > > not)
> > > > > > than
> > > > > > > an
> > > > > > > >> > > issue
> > > > > > > >> > > > in
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > > > > table
> > > > > > > >> > > > > > > > > > > > > > definition.
> > > > > > > >> > > > > > > > > > > > > > > > >> In fact, when using JPA,
> > > > separating
> > > > > > the
> > > > > > > >> > server
> > > > > > > >> > > > > model
> > > > > > > >> > > > > > > > from
> > > > > > > >> > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > JPA
> > > > > > > >> > > > > > > > > > > > > > > > entity is
> > > > > > > >> > > > > > > > > > > > > > > > >> not a good idea, especially
> > if
> > > > the
> > > > > > > entity
> > > > > > > >> > > > contains
> > > > > > > >> > > > > > > > > > collections.
> > > > > > > >> > > > > > > > > > > > I
> > > > > > > >> > > > > > > > > > > > > > will
> > > > > > > >> > > > > > > > > > > > > > > > try
> > > > > > > >> > > > > > > > > > > > > > > > >> to change that without
> > breaking
> > > > > > > anything.
> > > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > > >> > > > > > > > > > > > > > > > >> On Wed, Nov 22, 2023 at
> > > 12:10 PM
> > > > > > > Enrique
> > > > > > > >> > > > Gonzalez
> > > > > > > >> > > > > > > > > Martinez <
> > > > > > > >> > > > > > > > > > > > > > > > >> egonzalez@apache.org>
> wrote:
> > > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > > >> > > > > > > > > > > > > > > > >>> After the events split you
> > now
> > > > > will
> > > > > > > >> need to
> > > > > > > >> > > > > create
> > > > > > > >> > > > > > a
> > > > > > > >> > > > > > > > node
> > > > > > > >> > > > > > > > > > > > instance
> > > > > > > >> > > > > > > > > > > > > > > > >>> model instance of making
> > > > > independent
> > > > > > > >> from
> > > > > > > >> > the
> > > > > > > >> > > > > > process
> > > > > > > >> > > > > > > > > > instance.
> > > > > > > >> > > > > > > > > > > > > > > > >>> That should do the trick.
> > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > >> > > > > > > > > > > > > > > > >>> Regarding
> deleting/inserting
> > > it
> > > > > was
> > > > > > > >> fixed
> > > > > > > >> > at
> > > > > > > >> > > > some
> > > > > > > >> > > > > > > > point.
> > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > >> > > > > > > > > > > > > > > > >>> El mar, 21 nov 2023 a las
> > > 20:22,
> > > > > > > >> Francisco
> > > > > > > >> > > > Javier
> > > > > > > >> > > > > > > > Tirado
> > > > > > > >> > > > > > > > > > Sarti
> > > > > > > >> > > > > > > > > > > > > > > > >>> (<ft...@redhat.com>)
> > > > escribió:
> > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > >> > > > > > > > > > > > > > > > >>> > Hi Martin,
> > > > > > > >> > > > > > > > > > > > > > > > >>> > I have a task to review
> > > > > > performance
> > > > > > > of
> > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > ProcessInstanceNodeDataEventMerger
> > > > > > > >> > > > > > > > > > > > > > > > >>> > My idea is to reduce the
> > > > number
> > > > > of
> > > > > > > >> delete
> > > > > > > >> > > > > inserts
> > > > > > > >> > > > > > > > when
> > > > > > > >> > > > > > > > > > > > processing
> > > > > > > >> > > > > > > > > > > > > > > > >>> events
> > > > > > > >> > > > > > > > > > > > > > > > >>> > and try to do it
> > > incremental.
> > > > > > > >> > > > > > > > > > > > > > > > >>> > That should improve
> > > > performance.
> > > > > > > >> > > > > > > > > > > > > > > > >>> > PS:
> > > > > > > >> > > > > > > > > > > > > > > > >>> > I was planning to send
> an
> > > > e-mail
> > > > > > > >> tomorrow
> > > > > > > >> > > > > > > announcing
> > > > > > > >> > > > > > > > > > that in
> > > > > > > >> > > > > > > > > > > > > > case you
> > > > > > > >> > > > > > > > > > > > > > > > >>> were
> > > > > > > >> > > > > > > > > > > > > > > > >>> > already working on a fix
> > for
> > > > > > that. I
> > > > > > > >> > assume
> > > > > > > >> > > > you
> > > > > > > >> > > > > > are
> > > > > > > >> > > > > > > > not
> > > > > > > >> > > > > > > > > > and I
> > > > > > > >> > > > > > > > > > > > > > would
> > > > > > > >> > > > > > > > > > > > > > > > be
> > > > > > > >> > > > > > > > > > > > > > > > >>> > sending a PR soon.
> > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > >> > > > > > > > > > > > > > > > >>> > On Tue, Nov 21, 2023 at
> > > > 6:09 PM
> > > > > > > Martin
> > > > > > > >> > > Weiler
> > > > > > > >> > > > > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > >> > > > > > > > > > > > > > > > >>> > wrote:
> > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > I looked into the new
> > > > examples
> > > > > > > using
> > > > > > > >> > > > > data-index
> > > > > > > >> > > > > > > > > > persistence
> > > > > > > >> > > > > > > > > > > > > > addon -
> > > > > > > >> > > > > > > > > > > > > > > > >>> Neus'
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > PR#1813 [1] for
> > serverless
> > > > and
> > > > > > > >> Pere's
> > > > > > > >> > > > branch
> > > > > > > >> > > > > > [2]
> > > > > > > >> > > > > > > > for
> > > > > > > >> > > > > > > > > > > > workflow
> > > > > > > >> > > > > > > > > > > > > > > > (great
> > > > > > > >> > > > > > > > > > > > > > > > >>> job
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > both!) - and they work
> > > > without
> > > > > > > >> issues
> > > > > > > >> > > using
> > > > > > > >> > > > > > > single
> > > > > > > >> > > > > > > > > > > > requests.
> > > > > > > >> > > > > > > > > > > > > > > > >>> However, under
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > some load (I used 'ab'
> > for
> > > > > > testing
> > > > > > > >> > with a
> > > > > > > >> > > > > light
> > > > > > > >> > > > > > > > > > > > concurrency of
> > > > > > > >> > > > > > > > > > > > > > 10
> > > > > > > >> > > > > > > > > > > > > > > > >>> parallel
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > requests) I ran into
> the
> > > > > > following
> > > > > > > >> > > > problems:
> > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > (1) Large number of
> > > > > > insert/delete
> > > > > > > >> calls
> > > > > > > >> > > > (eg.
> > > > > > > >> > > > > > for
> > > > > > > >> > > > > > > > > tables
> > > > > > > >> > > > > > > > > > > > such as
> > > > > > > >> > > > > > > > > > > > > > > > >>> nodes,
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > definitions, etc)
> > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > (2) Hibernate
> > > > > > > >> OptimisticLockExceptions
> > > > > > > >> > /
> > > > > > > >> > > > > > > > > > > > StaleStateExceptions
> > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > (3) DB deadlocks
> > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > (4) Error responses,
> > slow
> > > > > > response
> > > > > > > >> > times
> > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > The reason I am
> reaching
> > > out
> > > > > > with
> > > > > > > >> this
> > > > > > > >> > > > topic
> > > > > > > >> > > > > > here
> > > > > > > >> > > > > > > > is
> > > > > > > >> > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > find
> > > > > > > >> > > > > > > > > > > > > > out if
> > > > > > > >> > > > > > > > > > > > > > > > >>> we are
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > aware of this issue,
> and
> > > if
> > > > > > > someone
> > > > > > > >> is
> > > > > > > >> > > > > already
> > > > > > > >> > > > > > > > > looking
> > > > > > > >> > > > > > > > > > > > into or
> > > > > > > >> > > > > > > > > > > > > > > > being
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > assigned to it?
> > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > Thanks,
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > Martin
> > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > [1]
> > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> >
> > > > https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > [2]
> > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > >
> > > > > > > >>
> > > > >
> ---------------------------------------------------------------------
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > To unsubscribe,
> e-mail:
> > > > > > > >> > > > > > > > > dev-unsubscribe@kie.apache.org
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > For additional
> commands,
> > > > > e-mail:
> > > > > > > >> > > > > > > > > > dev-help@kie.apache.org
> > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > >> > > > > > > > > > > > > > > > >>> To unsubscribe, e-mail:
> > > > > > > >> > > > > > > dev-unsubscribe@kie.apache.org
> > > > > > > >> > > > > > > > > > > > > > > > >>> For additional commands,
> > > e-mail:
> > > > > > > >> > > > > > > > dev-help@kie.apache.org
> > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > >
> > > > > > > >>
> > > > >
> ---------------------------------------------------------------------
> > > > > > > >> > > > > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > > > > > >> > > > > > dev-unsubscribe@kie.apache.org
> > > > > > > >> > > > > > > > > > > > > > > > > For additional commands,
> > e-mail:
> > > > > > > >> > > > > > > dev-help@kie.apache.org
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > >> > > > > > > > > > > > > > To unsubscribe, e-mail:
> > >
>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Enrique Gonzalez Martinez <eg...@apache.org>]

Hi Martin, the main problem regarding the purge is because it is still
unclear the policy for what tech to use and the future components.

Recently we had a discussion about proposing graphql for this sort of admin
tasks. So far for subsystems we have been using rest endpoints (like update
timers or modify human task or change processes). There is one exception
which is the gateway that is pure graphql and somehow uses graphql for
everything making complex operations under the hood.

This has somehow frozen the purge for data audit for a bit and the proposal
was to use rest endpoints to do the clean up in the component and offer the
graphql counterpart in the gateway promoting it to a first class citizen
component instead of having it embedded in the data index.

I would suggest to come up at least with a policy first regarding the
convention every component should address this.


El mar, 20 feb 2024, 23:39, Martin Weiler <mw...@ibm.com.invalid>
escribió:

> IMO, it is good to have this discussion around data sanity now instead of
> putting it off until later when data has already accumulated in production
> environments.
>
> Based on the input here, we are dealing with three types of data:
> 1. Runtime data - active instances only, engine cleans up the data
> automatically at process instance end
> 2. Historic log data - data created by data-audit intended for long term
> storage
> 3. Data-index data - somehow this data falls in between the two
> aforementioned categories, with the idea of the data being "recent", but
> not restricted to active instances only
>
> We'd need purge strategies for both #2 and #3 (perhaps different ones, or
> with different config settings) in order to prevent unlimited data growth.
>
> ________________________________________
> From: Enrique Gonzalez Martinez <eg...@apache.org>
> Sent: Monday, February 19, 2024 7:11 AM
> To: dev@kie.apache.org
> Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with data-index
> persistence addon
>
> Hi Francisco,
> To give you more context about this.
>
> STP is a concept, a process with certain constraints: no persistence and
> returning the outcome in the call (sync execution with no idle states). It
> was a requirement from a user in the past. One of the requirements was
> leaving no trail. In v7 was easy because you could disable the audit in
> that case. Actually we have the same way to do what we did in v7 in here as
> you can add/remove index just removing deps.
>
> We have the same outcome with different approaches and STP is already
> delivered.
>
> El lun, 19 feb 2024 a las 14:46, Francisco Javier Tirado Sarti (<
> ftirados@redhat.com>) escribió:
>
> > Regarding STP (which is not a concept that we have in the code. I mean
> STP
> > are processes as nonSTP are), I guess, as all processes, they were kept
> in
> > DataIndex once completed because users wanted (and still wants) to check
> > the result once the call had been performed. If we want to leave no trace
> > of them in DataIndex for some reason, we will need to make it a
> > Runtimes concept so DataIndex can handle them in a different way.
> >
> > On Mon, Feb 19, 2024 at 2:27 PM Enrique Gonzalez Martinez <
> > egonzalez@apache.org> wrote:
> >
> > > Alex:
> > > Right now the data index is working in the same way as it did in v7
> with
> > > the emitters. The only difference between two impl is that in here the
> > > storage is pgsql instead elastic search.  You are right regarding is a
> > > snapshot of the last state of the process but we did never define how
> > long
> > > would be alive that dats Honestly i am happy right now with the way it
> > > works. The clean up mechanism is still tbd because we still need to
> > discuss
> > > other stuff first.
> > >
> > >
> > > Regarding stp is to leave no trail because u can get the outcome
> directly
> > > from the call. It was defined like that in v7. So there is no use for
> the
> > > index or the audit.
> > >
> > > El lun, 19 feb 2024, 14:13, Francisco Javier Tirado Sarti <
> > > ftirados@redhat.com> escribió:
> > >
> > > > Hi Alex,
> > > > There has been some confusion about the purpose of DataIndex. To be
> > > honest
> > > > I believe they were already sorted out, but your e-mail makes me
> think
> > > that
> > > > is not the case ;). I let Kris to clarify that with you. My view is
> > that
> > > > data-index is a way to query recently closed and active processes
> (the
> > > key
> > > > here is the definition of recently, which in my opinion should be
> > > > configurable)
> > > > But, besides that discussion and being pragmatic, keeping finishing
> > > process
> > > > instances "for a while" in DataIndex was the only way for users to
> > query
> > > > the result of straight through processes. That's a function that
> cannot
> > > be
> > > > removed right now
> > > >
> > > > On Mon, Feb 19, 2024 at 1:33 PM Alex Porcelli <po...@apache.org>
> > > wrote:
> > > >
> > > > > if data index was supposed to provide snapshot view of the process
> > > > > instance… why do we keep it after the process instance is finished?
> > > > >
> > > > >
> > > > > On Mon, Feb 19, 2024 at 7:12 AM Francisco Javier Tirado Sarti <
> > > > > ftirados@redhat.com> wrote:
> > > > >
> > > > > > Hi Martin.
> > > > > > After taking a deeper look at this, I realize that the behaviour
> is
> > > the
> > > > > > expected one.
> > > > > > Runtimes DB does not track the completed process instance (that's
> > > what
> > > > > the
> > > > > > JDBCProcessInstances warn is telling us), but DataIndex, as
> > expected,
> > > > is
> > > > > > tracking it in processes and nodes table. And yes it will grow
> over
> > > > time.
> > > > > > What we need is some configurable purge mechanism for DataIndex,
> so
> > > it
> > > > > > eventually removes older completed process instances.
> > > > > >
> > > > > > On Tue, Feb 13, 2024 at 12:59 PM Francisco Javier Tirado Sarti <
> > > > > > ftirados@redhat.com> wrote:
> > > > > >
> > > > > > > Hi Martin,
> > > > > > > Good catch!. Looks like the skipping performed for process
> > > instances
> > > > is
> > > > > > > not applied to node instances. Something we definitely need to
> > > review
> > > > > on
> > > > > > > the runtimes side.
> > > > > > >
> > > > > > > On Mon, Feb 12, 2024 at 11:59 PM Martin Weiler
> > > > <mweiler@ibm.com.invalid
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> On a somewhat related note, testing a simple workflow (start
> ->
> > > > script
> > > > > > >> node -> end), I see the following messages in the logs:
> > > > > > >> 2024-02-12 22:49:50,493 28758dde544c WARN
> > > > > > >> [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> > > > > > >> (executor-thread-3) Skipping create of process instance id:
> > > > > > >> 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2
> > > > > > >>
> > > > > > >> So far, so good. And I'd expect to see no trace of this
> process
> > in
> > > > the
> > > > > > >> database if I don't have data audit enabled.
> > > > > > >>
> > > > > > >> However, the 'processes' table contains a row with state=2,
> with
> > > > > related
> > > > > > >> entries in the 'nodes' table. In a load test, I see these
> tables
> > > > grow
> > > > > > >> significantly over time. Am I missing something to have these
> > > > entries
> > > > > > >> cleaned up automatically?
> > > > > > >>
> > > > > > >> ________________________________________
> > > > > > >> From: Martin Weiler <mw...@ibm.com.INVALID>
> > > > > > >> Sent: Monday, February 12, 2024 3:40 PM
> > > > > > >> To: dev@kie.apache.org
> > > > > > >> Subject: [EXTERNAL] RE: [DISCUSSION] Performance issues with
> > > > > data-index
> > > > > > >> persistence addon
> > > > > > >>
> > > > > > >> Thanks everyone for your input. Based on this discussion, I
> > opened
> > > > the
> > > > > > >> following PR:
> > > > > > >> https://github.com/apache/incubator-kie-kogito-apps/pull/1985
> > > > > > >>
> > > > > > >> With this change, the performance seems to be stable over
> time:
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
> > > > > > >>
> > > > > > >> Martin
> > > > > > >>
> > > > > > >> ________________________________________
> > > > > > >> From: Gonzalo Muñoz <gm...@apache.org>
> > > > > > >> Sent: Friday, February 9, 2024 9:42 AM
> > > > > > >> To: dev@kie.apache.org
> > > > > > >> Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > > > > data-index
> > > > > > >> persistence addon
> > > > > > >>
> > > > > > >> Great work Francisco,
> > > > > > >> Martin, take a look at this link with some related tips (in
> case
> > > you
> > > > > > find
> > > > > > >> it useful):
> > > > > > >>
> https://www.cybertec-postgresql.com/en/index-your-foreign-key/
> > > > > > >>
> > > > > > >> El vie, 9 feb 2024 a las 17:20, Francisco Javier Tirado Sarti
> (<
> > > > > > >> ftirados@redhat.com>) escribió:
> > > > > > >>
> > > > > > >> > For the moment being, we will keep JPA till we exhaust all
> > > > > > >> possibilities,
> > > > > > >> > let's call switching from jpa to jdbc our hidden plan B ;)
> > > > > > >> > I already told Martin, but in order everyone to know, just
> > after
> > > > > > writing
> > > > > > >> > the previous email, I thought "what if Postgres is not
> > > > automatically
> > > > > > >> > indexing foreign keys like mysql?" and, eureka
> > > > > > >> > Postgres doc
> > > > > > >> >
> https://www.postgresql.org/docs/current/ddl-constraints.html
> > > > > > >> > Mysql doc
> > > > > > >> >
> > > > https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> > > > > > >> > These are the relevant excerpt
> > > > > > >> >
> > > > > > >> > *Postgresql*
> > > > > > >> > *A foreign key must reference columns that either are a
> > primary
> > > > key
> > > > > or
> > > > > > >> form
> > > > > > >> > a unique constraint, or are columns from a non-partial
> unique
> > > > index.
> > > > > > >> This
> > > > > > >> > means that the referenced columns always have an index to
> > allow
> > > > > > >> efficient
> > > > > > >> > lookups on whether a referencing row has a match. Since a
> > DELETE
> > > > of
> > > > > a
> > > > > > >> row
> > > > > > >> > from the referenced table or an UPDATE of a referenced
> column
> > > will
> > > > > > >> require
> > > > > > >> > a scan of the referencing table for rows matching the old
> > value,
> > > > it
> > > > > is
> > > > > > >> > often a good idea to index the referencing columns too.
> > Because
> > > > this
> > > > > > is
> > > > > > >> not
> > > > > > >> > always needed, and there are many choices available on how
> to
> > > > index,
> > > > > > the
> > > > > > >> > declaration of a foreign key constraint does not
> automatically
> > > > > create
> > > > > > an
> > > > > > >> > index on the referencing columns.*
> > > > > > >> > *Mysql*
> > > > > > >> > *MySQL requires that foreign key columns be indexed; if you
> > > > create a
> > > > > > >> table
> > > > > > >> > with a foreign key constraint but no index on a given
> column,
> > an
> > > > > index
> > > > > > >> is
> > > > > > >> > created. *
> > > > > > >> >
> > > > > > >> > So I asked Martin to especially create an index for
> > > > > > process_instance_id
> > > > > > >> > column on nodes table
> > > > > > >> > I think that will fix the problem detected on the thread
> dump.
> > > > > > >> > The simpler process test to verify queries are fine still
> > > stands,
> > > > > > >> though ;)
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <
> > > tzimanyi@apache.org
> > > > >
> > > > > > >> wrote:
> > > > > > >> >
> > > > > > >> > > I always preferred pure JDBC over Hibernate myself, just
> for
> > > the
> > > > > > sake
> > > > > > >> of
> > > > > > >> > > control of what is happening :) So I would not -1 that
> > myself.
> > > > > > >> > >
> > > > > > >> > > Tibor
> > > > > > >> > >
> > > > > > >> > > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado Sarti <
> > > > > > >> > > ftirados@redhat.com>
> > > > > > >> > > napísal(a):
> > > > > > >> > >
> > > > > > >> > > > Hi,
> > > > > > >> > > > Usually I do not want to talk about work in progress
> > because
> > > > > > >> > preliminary
> > > > > > >> > > > conclusions are pretty volatile but, well, there are a
> > > couple
> > > > of
> > > > > > >> things
> > > > > > >> > > > that can be concluded from the really valuable
> information
> > > > that
> > > > > > >> Martin
> > > > > > >> > > > provided.
> > > > > > >> > > > 1) In order to be able to determine if the number of
> > > > statements
> > > > > is
> > > > > > >> > larger
> > > > > > >> > > > than expected, I asked Martin to test with a simpler
> > process
> > > > > > >> > definition.
> > > > > > >> > > > One with just three nodes: start, script and end. The
> > script
> > > > one
> > > > > > >> should
> > > > > > >> > > > change just one variable. This way we can analyze if the
> > > > number
> > > > > of
> > > > > > >> > > queries
> > > > > > >> > > > is the expected one. From the single log (audit was
> > > activated
> > > > > > them)
> > > > > > >> my
> > > > > > >> > > > conclusion is that the number of insert/updates over
> > > processes
> > > > > and
> > > > > > >> > nodes
> > > > > > >> > > > (there a lot over task, that I will prefer to skip for
> > now,
> > > > baby
> > > > > > >> steps)
> > > > > > >> > > is
> > > > > > >> > > > the expected one.
> > > > > > >> > > > 2) Analysing the thread dump, we see around 15 threads
> > > > executing
> > > > > > >> this
> > > > > > >> > > line
> > > > > > >> > > > at
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > > > > >> > > > so its pretty clear the code to be optimized ;). I'm
> > > > evaluating
> > > > > > >> > > > possibilities within JPA/Hibernate, but I'm starting to
> > > think
> > > > > that
> > > > > > >> it
> > > > > > >> > > might
> > > > > > >> > > > be better to switch to JDBC and skip hibernate. Our
> lives
> > > will
> > > > > be
> > > > > > >> > > simpler,
> > > > > > >> > > > especially with a schema relatively simple like ours
> (that
> > > > will
> > > > > be
> > > > > > >> my
> > > > > > >> > > > recommendation if I was an external consultant)
> > > > > > >> > > >
> > > > > > >> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <
> > > > > tzimanyi@apache.org
> > > > > > >
> > > > > > >> > > wrote:
> > > > > > >> > > >
> > > > > > >> > > > > Hi,
> > > > > > >> > > > >
> > > > > > >> > > > > this will be a bit off-topic. However as far as
> > > > performance, I
> > > > > > >> think
> > > > > > >> > we
> > > > > > >> > > > > should think about that we have string primary keys
> > > (IDs). I
> > > > > > would
> > > > > > >> > > expect
> > > > > > >> > > > > the database systems are much better with indexing
> > numeric
> > > > > keys
> > > > > > >> than
> > > > > > >> > > > > strings. I remember from the past, when I was working
> > with
> > > > > DBs,
> > > > > > >> that
> > > > > > >> > > > using
> > > > > > >> > > > > strings as keys or indexes was a discouraged practice.
> > > > > > >> > > > >
> > > > > > >> > > > > Best regards,
> > > > > > >> > > > > Tibor
> > > > > > >> > > > >
> > > > > > >> > > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler
> > > > > <mweiler@ibm.com.invalid
> > > > > > >
> > > > > > >> > > > > napísal(a):
> > > > > > >> > > > >
> > > > > > >> > > > > > I changed the test to use MongoDB [1] and I don't
> see
> > a
> > > > > > >> performance
> > > > > > >> > > > > > degradation with this setup [2].
> > > > > > >> > > > > >
> > > > > > >> > > > > > Please keep us posted of your findings. Thanks!
> > > > > > >> > > > > >
> > > > > > >> > > > > > Martin
> > > > > > >> > > > > >
> > > > > > >> > > > > > [1]
> > > > > > >> > > > >
> > > > > > >> >
> > > > > >
> > > https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > > > >> > > > > > [2]
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > > > >> > > > > >
> > > > > > >> > > > > > ________________________________________
> > > > > > >> > > > > > From: Francisco Javier Tirado Sarti <
> > > ftirados@redhat.com>
> > > > > > >> > > > > > Sent: Wednesday, February 7, 2024 11:40 AM
> > > > > > >> > > > > > To: dev@kie.apache.org
> > > > > > >> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance
> > issues
> > > > with
> > > > > > >> > > data-index
> > > > > > >> > > > > > persistence addon
> > > > > > >> > > > > >
> > > > > > >> > > > > > yes, it can be index degradation because of size,
> but
> > I
> > > > > > believe
> > > > > > >> (I
> > > > > > >> > > > might
> > > > > > >> > > > > be
> > > > > > >> > > > > > wrong) the db is too small (yet) for that.
> > > > > > >> > > > > > But, eventually, Postgres, when the DB is huge
> enough,
> > > > > > >> unavoidably
> > > > > > >> > > will
> > > > > > >> > > > > > behave like the graphic that Martin sent.
> > > > > > >> > > > > > Since I believe we are not huge enough (yet), lets
> > rule
> > > > out
> > > > > > >> another
> > > > > > >> > > > issue
> > > > > > >> > > > > > by analysing the sql logs (I requested those to
> Martin
> > > > > offline
> > > > > > >> and
> > > > > > >> > he
> > > > > > >> > > > is
> > > > > > >> > > > > > going to kindly collect them).
> > > > > > >> > > > > > Also Im curious to know if Mongo behave in the same
> > way.
> > > > > > >> > > > > >
> > > > > > >> > > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez
> > > Martinez <
> > > > > > >> > > > > > egonzalez@apache.org> wrote:
> > > > > > >> > > > > >
> > > > > > >> > > > > > > Hi Francisco,
> > > > > > >> > > > > > > I would highly recommend to check indexes and how
> > the
> > > > > > updates
> > > > > > >> > work
> > > > > > >> > > in
> > > > > > >> > > > > > data
> > > > > > >> > > > > > > index to avoid full scan table and lock the full
> > > table.
> > > > > Some
> > > > > > >> db
> > > > > > >> > are
> > > > > > >> > > > > very
> > > > > > >> > > > > > > sensitive to that.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > El mié, 7 feb 2024, 18:41, Francisco Javier Tirado
> > > > Sarti <
> > > > > > >> > > > > > > ftirados@redhat.com> escribió:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > Hi Martin,
> > > > > > >> > > > > > > > While I analyze the data, let me ask you if it
> is
> > > > > possible
> > > > > > >> to
> > > > > > >> > > > perform
> > > > > > >> > > > > > > > another check (similar in a way to disabling
> > > > data-index
> > > > > > like
> > > > > > >> > you
> > > > > > >> > > > do)
> > > > > > >> > > > > > Can
> > > > > > >> > > > > > > > you switch to MongoDB persistence and check if
> the
> > > > same
> > > > > > >> > > degradation
> > > > > > >> > > > > > that
> > > > > > >> > > > > > > is
> > > > > > >> > > > > > > > there for postgres remains?
> > > > > > >> > > > > > > > I do not know if this is feasible but will
> > certainly
> > > > > > >> indicate
> > > > > > >> > the
> > > > > > >> > > > > > problem
> > > > > > >> > > > > > > > is on the postgres storage layer and I do not
> > have a
> > > > > clear
> > > > > > >> > > > prediction
> > > > > > >> > > > > > of
> > > > > > >> > > > > > > > what we will see when doing this switch.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler
> > > > > > >> > > > <mweiler@ibm.com.invalid
> > > > > > >> > > > > >
> > > > > > >> > > > > > > > wrote:
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > > Hi Francisco,
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > thanks for your work on this important topic!
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > I would like to share some test results here,
> > > which
> > > > > > might
> > > > > > >> > help
> > > > > > >> > > to
> > > > > > >> > > > > > > improve
> > > > > > >> > > > > > > > > the codebase even further. I am using the
> jmeter
> > > > based
> > > > > > >> test
> > > > > > >> > > case
> > > > > > >> > > > > from
> > > > > > >> > > > > > > > Pere
> > > > > > >> > > > > > > > > and Enrique (thanks guys!) [1] which uses a
> load
> > > of
> > > > 30
> > > > > > >> > threads
> > > > > > >> > > to
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > 1) start a new process instance (POST)
> > > > > > >> > > > > > > > > 2) retrieve tasks for a user (GET)
> > > > > > >> > > > > > > > > 3) fetches task details (GET)
> > > > > > >> > > > > > > > > 4) complete a task (POST)
> > > > > > >> > > > > > > > > 5) execute a query on data-audit
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > With this test setup, I noticed that the
> > > performance
> > > > > for
> > > > > > >> the
> > > > > > >> > > POST
> > > > > > >> > > > > > > > > requests, in particular the one to start a new
> > > > process
> > > > > > >> > > instance,
> > > > > > >> > > > > > > degrades
> > > > > > >> > > > > > > > > over time - see graph [2]. If I run the same
> > test
> > > > > > without
> > > > > > >> > > > > data-index,
> > > > > > >> > > > > > > > then
> > > > > > >> > > > > > > > > there is no such performance degradation [3].
> > You
> > > > can
> > > > > > >> find a
> > > > > > >> > > > thread
> > > > > > >> > > > > > > dump
> > > > > > >> > > > > > > > > captured a few minutes into the first test
> here
> > > [4]
> > > > > that
> > > > > > >> > might
> > > > > > >> > > > help
> > > > > > >> > > > > > to
> > > > > > >> > > > > > > > see
> > > > > > >> > > > > > > > > some of the contention points.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > I'd appreciate if you could take a look and
> see
> > if
> > > > > there
> > > > > > >> is
> > > > > > >> > > > > something
> > > > > > >> > > > > > > > that
> > > > > > >> > > > > > > > > can be further improved based on your previous
> > > work.
> > > > > If
> > > > > > >> you
> > > > > > >> > > need
> > > > > > >> > > > > any
> > > > > > >> > > > > > > > > additional data, let me know, but otherwise it
> > is
> > > > > > >> > > straightforward
> > > > > > >> > > > > to
> > > > > > >> > > > > > > run
> > > > > > >> > > > > > > > > the jmeter test as well.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Thanks,
> > > > > > >> > > > > > > > > Martin
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > [1]
> > > > > > >> https://github.com/pefernan/job-service-refactor-test/
> > > > > > >> > > > > > > > > [2]
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > > >> > > > > > > > > [3]
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > > >> > > > > > > > > [4]
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > ________________________________________
> > > > > > >> > > > > > > > > From: Francisco Javier Tirado Sarti <
> > > > > > ftirados@redhat.com>
> > > > > > >> > > > > > > > > Sent: Wednesday, January 17, 2024 9:13 AM
> > > > > > >> > > > > > > > > To: dev@kie.apache.org
> > > > > > >> > > > > > > > > Cc: Pere Fernandez Perez
> > > > > > >> > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION]
> Performance
> > > > > issues
> > > > > > >> with
> > > > > > >> > > > > > data-index
> > > > > > >> > > > > > > > > persistence addon
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Hi Alex,
> > > > > > >> > > > > > > > > I did not take times (which depends on a
> number
> > of
> > > > > > >> variables
> > > > > > >> > > that
> > > > > > >> > > > > > > > > drastically change between environments), but
> > > verify
> > > > > > that
> > > > > > >> the
> > > > > > >> > > > > number
> > > > > > >> > > > > > of
> > > > > > >> > > > > > > > > updates has been reduced drastically without
> > > losing
> > > > > > >> > > > functionality,
> > > > > > >> > > > > > > which
> > > > > > >> > > > > > > > is
> > > > > > >> > > > > > > > > objectively a good thing. If before the
> change,
> > > for
> > > > > > every
> > > > > > >> > node
> > > > > > >> > > > > > > executed,
> > > > > > >> > > > > > > > we
> > > > > > >> > > > > > > > > have an update for every node previously
> > executed,
> > > > so
> > > > > > if a
> > > > > > >> > > > process
> > > > > > >> > > > > > have
> > > > > > >> > > > > > > > 50
> > > > > > >> > > > > > > > > nodes to execute, we were performing nearly
> > > 50*51/2
> > > > > > >> updates,
> > > > > > >> > > > which
> > > > > > >> > > > > > > gives
> > > > > > >> > > > > > > > us
> > > > > > >> > > > > > > > > a total of  1275 updates, now we have just one
> > for
> > > > > every
> > > > > > >> node
> > > > > > >> > > > being
> > > > > > >> > > > > > > > > executed, implying a total of 50 updates.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli
> <
> > > > > > >> > > alex@porcelli.me>
> > > > > > >> > > > > > > wrote:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > Francisco,
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > I noticed that your PR has been merged, but
> I
> > > was
> > > > > > >> expecting
> > > > > > >> > > (at
> > > > > > >> > > > > > least
> > > > > > >> > > > > > > > > > was my understanding from this thread) that
> > > before
> > > > > > >> merging
> > > > > > >> > > some
> > > > > > >> > > > > > > > > > benchmark data would be shared in advance -
> to
> > > > > assess
> > > > > > >> the
> > > > > > >> > > > > > > cost/benefit
> > > > > > >> > > > > > > > > > of such a decent size change.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Do you have any information to share?
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM Francisco
> > Javier
> > > > > > Tirado
> > > > > > >> > Sarti
> > > > > > >> > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > Yes, as intended, now we have one select
> and
> > > one
> > > > > > >> > > > insert/update
> > > > > > >> > > > > > per
> > > > > > >> > > > > > > > node
> > > > > > >> > > > > > > > > > > event.
> > > > > > >> > > > > > > > > > > I moved the PR as ready for review and
> give
> > > > @Pere
> > > > > > >> > Fernandez
> > > > > > >> > > > > Perez
> > > > > > >> > > > > > > > > > > <pe...@redhat.com> permission to the
> > > branch
> > > > so
> > > > > > he
> > > > > > >> can
> > > > > > >> > > > edit
> > > > > > >> > > > > it
> > > > > > >> > > > > > > in
> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > > > > > > next two weeks (Ill be on PTO)  if
> desired,
> > > > before
> > > > > > >> > merging.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex
> > Porcelli
> > > <
> > > > > > >> > > > > alex@porcelli.me>
> > > > > > >> > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > Cool, thank you Francisco!
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > Did you manage to get some preliminary
> > data
> > > > > about
> > > > > > >> > > > > improvements?
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM
> Francisco
> > > > > Javier
> > > > > > >> > Tirado
> > > > > > >> > > > > Sarti
> > > > > > >> > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > Yes, after some delay because of
> > quarkus 3
> > > > > > >> migration.
> > > > > > >> > > Im
> > > > > > >> > > > > > > refining
> > > > > > >> > > > > > > > > > this
> > > > > > >> > > > > > > > > > > > > draft PR
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > >
> > > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex
> > > > Porcelli
> > > > > <
> > > > > > >> > > > > > > alex@porcelli.me>
> > > > > > >> > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > Any update or new findings on this
> > > topic?
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM
> > > Francisco
> > > > > > Javier
> > > > > > >> > > Tirado
> > > > > > >> > > > > > Sarti
> > > > > > >> > > > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > Hi Alex,
> > > > > > >> > > > > > > > > > > > > > > After considering different
> options
> > to
> > > > > > improve
> > > > > > >> > > > > > performance,
> > > > > > >> > > > > > > > we
> > > > > > >> > > > > > > > > > feel
> > > > > > >> > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > > it
> > > > > > >> > > > > > > > > > > > > > > is time to "partially" move away
> > from
> > > > the
> > > > > > >> current
> > > > > > >> > > Map
> > > > > > >> > > > > > style
> > > > > > >> > > > > > > > > > > > interface (
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > >> > > > > > > > > > > > > > )
> > > > > > >> > > > > > > > > > > > > > > which was shared with Trusty, to
> one
> > > > more
> > > > > > >> > suitable
> > > > > > >> > > > for
> > > > > > >> > > > > > > usage
> > > > > > >> > > > > > > > > > with a
> > > > > > >> > > > > > > > > > > > > > > relational DB like postgresql (but
> > > still
> > > > > > >> > compatible
> > > > > > >> > > > > with
> > > > > > >> > > > > > > big
> > > > > > >> > > > > > > > > > table
> > > > > > >> > > > > > > > > > > > dbs).
> > > > > > >> > > > > > > > > > > > > > > The idea will be to replace
> generic
> > > > > Storage
> > > > > > >> > > interface
> > > > > > >> > > > > by
> > > > > > >> > > > > > > four
> > > > > > >> > > > > > > > > > > > specific
> > > > > > >> > > > > > > > > > > > > > > interfaces (which will inherit
> from
> > a
> > > > > common
> > > > > > >> one
> > > > > > >> > > that
> > > > > > >> > > > > > keeps
> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > > > > > query
> > > > > > >> > > > > > > > > > > > > > part
> > > > > > >> > > > > > > > > > > > > > > at is it. with get and query
> > methods),
> > > > > that
> > > > > > >> will
> > > > > > >> > > > > include
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > > > required
> > > > > > >> > > > > > > > > > > > > > > modification operations for the
> four
> > > > > > DataIndex
> > > > > > >> > > > > "domains":
> > > > > > >> > > > > > > > > > > > > > processinstance,
> > > > > > >> > > > > > > > > > > > > > > usertask, processdefinitions and
> > jobs.
> > > > > Those
> > > > > > >> > > > interfaces
> > > > > > >> > > > > > > will
> > > > > > >> > > > > > > > > > define
> > > > > > >> > > > > > > > > > > > > > methods
> > > > > > >> > > > > > > > > > > > > > > like addNode, addVariable,
> > updateTask,
> > > > > > >> > > > > addAttachment.....
> > > > > > >> > > > > > > > that
> > > > > > >> > > > > > > > > > will
> > > > > > >> > > > > > > > > > > > allow
> > > > > > >> > > > > > > > > > > > > > > the persistent layer
> implementation
> > > to
> > > > > just
> > > > > > >> > update
> > > > > > >> > > > the
> > > > > > >> > > > > > > > needed
> > > > > > >> > > > > > > > > > info
> > > > > > >> > > > > > > > > > > > in
> > > > > > >> > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > DB  (for example, for addNode in
> > > > Postgres,
> > > > > > >> just
> > > > > > >> > > > insert
> > > > > > >> > > > > a
> > > > > > >> > > > > > > row
> > > > > > >> > > > > > > > > into
> > > > > > >> > > > > > > > > > > > nodes
> > > > > > >> > > > > > > > > > > > > > > table, for addNode in Mongo,
> > basically
> > > > the
> > > > > > >> same
> > > > > > >> > > > atomic
> > > > > > >> > > > > > > upsert
> > > > > > >> > > > > > > > > > > > operation
> > > > > > >> > > > > > > > > > > > > > > that is currently done).
> Therefore,
> > we
> > > > > > >> increase
> > > > > > >> > > > > > performance
> > > > > > >> > > > > > > > for
> > > > > > >> > > > > > > > > > > > Postgres
> > > > > > >> > > > > > > > > > > > > > > and keep the current one for
> Mongo.
> > > The
> > > > > > >> current
> > > > > > >> > DB
> > > > > > >> > > > > > schemas
> > > > > > >> > > > > > > > > won't
> > > > > > >> > > > > > > > > > be
> > > > > > >> > > > > > > > > > > > > > > touched.
> > > > > > >> > > > > > > > > > > > > > > Since the code change is large, I
> do
> > > not
> > > > > > think
> > > > > > >> > I'll
> > > > > > >> > > > be
> > > > > > >> > > > > > able
> > > > > > >> > > > > > > > to
> > > > > > >> > > > > > > > > > have
> > > > > > >> > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > PR
> > > > > > >> > > > > > > > > > > > > > > ready till next week.
> > > > > > >> > > > > > > > > > > > > > > But before starting, please let me
> > > know
> > > > if
> > > > > > >> that
> > > > > > >> > > > > approach
> > > > > > >> > > > > > is
> > > > > > >> > > > > > > > > fine
> > > > > > >> > > > > > > > > > for
> > > > > > >> > > > > > > > > > > > you.
> > > > > > >> > > > > > > > > > > > > > > Best regards.
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 6:55 PM
> Alex
> > > > > > Porcelli
> > > > > > >> <
> > > > > > >> > > > > > > > > alex@porcelli.me>
> > > > > > >> > > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > Thank you Francisco to getting
> > > deeper
> > > > on
> > > > > > >> this…
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > Looking forward to see the
> results
> > > of
> > > > > your
> > > > > > >> > > > suggested
> > > > > > >> > > > > > > > > > improvements.
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 9:40 AM
> > > > > Francisco
> > > > > > >> > Javier
> > > > > > >> > > > > Tirado
> > > > > > >> > > > > > > > > Sarti <
> > > > > > >> > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > I forgot to attach the queries
> > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> 3:04 PM
> > > > > > Francisco
> > > > > > >> > > Javier
> > > > > > >> > > > > > Tirado
> > > > > > >> > > > > > > > > > Sarti <
> > > > > > >> > > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > >> Hi,
> > > > > > >> > > > > > > > > > > > > > > > >> A brief update on this topic.
> > > > > > >> > > > > > > > > > > > > > > > >> After doing a simple test
> with
> > > > > example
> > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > >> > > > > > > > > > > > > > > > ,
> > > > > > >> > > > > > > > > > > > > > > > >> the number of updates over
> > Nodes
> > > > > table
> > > > > > is
> > > > > > >> > n*n,
> > > > > > >> > > > so
> > > > > > >> > > > > we
> > > > > > >> > > > > > > > > manage
> > > > > > >> > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > obtain a
> > > > > > >> > > > > > > > > > > > > > > > >> perfect quadratic performance
> > > > > > >> degradation.
> > > > > > >> > The
> > > > > > >> > > > > > problem
> > > > > > >> > > > > > > > is
> > > > > > >> > > > > > > > > > worse
> > > > > > >> > > > > > > > > > > > in
> > > > > > >> > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > case
> > > > > > >> > > > > > > > > > > > > > > > >> of Serverless Workflow than
> in
> > > BPMN
> > > > > > >> because
> > > > > > >> > we
> > > > > > >> > > > the
> > > > > > >> > > > > > > > number
> > > > > > >> > > > > > > > > of
> > > > > > >> > > > > > > > > > > > nodes
> > > > > > >> > > > > > > > > > > > > > is
> > > > > > >> > > > > > > > > > > > > > > > >> greater than the number of
> > > states.
> > > > In
> > > > > > >> that
> > > > > > >> > > > > example N
> > > > > > >> > > > > > > is
> > > > > > >> > > > > > > > > 16,
> > > > > > >> > > > > > > > > > but
> > > > > > >> > > > > > > > > > > > for
> > > > > > >> > > > > > > > > > > > > > a
> > > > > > >> > > > > > > > > > > > > > > > more
> > > > > > >> > > > > > > > > > > > > > > > >> complex workflow it would be
> > > > > certainly
> > > > > > >> > large.
> > > > > > >> > > > > > > > > > > > > > > > >> I think that this is more
> > related
> > > > to
> > > > > > how
> > > > > > >> we
> > > > > > >> > > are
> > > > > > >> > > > > > > handling
> > > > > > >> > > > > > > > > > JPA in
> > > > > > >> > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > code,
> > > > > > >> > > > > > > > > > > > > > > > >> in particular the mapping
> from
> > > > model
> > > > > to
> > > > > > >> > entity
> > > > > > >> > > > > > > > (basically
> > > > > > >> > > > > > > > > > JPA is
> > > > > > >> > > > > > > > > > > > > > blind
> > > > > > >> > > > > > > > > > > > > > > > and
> > > > > > >> > > > > > > > > > > > > > > > >> has to update all nodes for
> > every
> > > > > write
> > > > > > >> > > because
> > > > > > >> > > > it
> > > > > > >> > > > > > > > > believes
> > > > > > >> > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > node has
> > > > > > >> > > > > > > > > > > > > > > > >> been updated, although it is
> > not)
> > > > > than
> > > > > > an
> > > > > > >> > > issue
> > > > > > >> > > > in
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > > > table
> > > > > > >> > > > > > > > > > > > > > definition.
> > > > > > >> > > > > > > > > > > > > > > > >> In fact, when using JPA,
> > > separating
> > > > > the
> > > > > > >> > server
> > > > > > >> > > > > model
> > > > > > >> > > > > > > > from
> > > > > > >> > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > JPA
> > > > > > >> > > > > > > > > > > > > > > > entity is
> > > > > > >> > > > > > > > > > > > > > > > >> not a good idea, especially
> if
> > > the
> > > > > > entity
> > > > > > >> > > > contains
> > > > > > >> > > > > > > > > > collections.
> > > > > > >> > > > > > > > > > > > I
> > > > > > >> > > > > > > > > > > > > > will
> > > > > > >> > > > > > > > > > > > > > > > try
> > > > > > >> > > > > > > > > > > > > > > > >> to change that without
> breaking
> > > > > > anything.
> > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > > > > > >> On Wed, Nov 22, 2023 at
> > 12:10 PM
> > > > > > Enrique
> > > > > > >> > > > Gonzalez
> > > > > > >> > > > > > > > > Martinez <
> > > > > > >> > > > > > > > > > > > > > > > >> egonzalez@apache.org> wrote:
> > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > > > > > >>> After the events split you
> now
> > > > will
> > > > > > >> need to
> > > > > > >> > > > > create
> > > > > > >> > > > > > a
> > > > > > >> > > > > > > > node
> > > > > > >> > > > > > > > > > > > instance
> > > > > > >> > > > > > > > > > > > > > > > >>> model instance of making
> > > > independent
> > > > > > >> from
> > > > > > >> > the
> > > > > > >> > > > > > process
> > > > > > >> > > > > > > > > > instance.
> > > > > > >> > > > > > > > > > > > > > > > >>> That should do the trick.
> > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > > > >>> Regarding deleting/inserting
> > it
> > > > was
> > > > > > >> fixed
> > > > > > >> > at
> > > > > > >> > > > some
> > > > > > >> > > > > > > > point.
> > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > > > >>> El mar, 21 nov 2023 a las
> > 20:22,
> > > > > > >> Francisco
> > > > > > >> > > > Javier
> > > > > > >> > > > > > > > Tirado
> > > > > > >> > > > > > > > > > Sarti
> > > > > > >> > > > > > > > > > > > > > > > >>> (<ft...@redhat.com>)
> > > escribió:
> > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > >> > > > > > > > > > > > > > > > >>> > Hi Martin,
> > > > > > >> > > > > > > > > > > > > > > > >>> > I have a task to review
> > > > > performance
> > > > > > of
> > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > ProcessInstanceNodeDataEventMerger
> > > > > > >> > > > > > > > > > > > > > > > >>> > My idea is to reduce the
> > > number
> > > > of
> > > > > > >> delete
> > > > > > >> > > > > inserts
> > > > > > >> > > > > > > > when
> > > > > > >> > > > > > > > > > > > processing
> > > > > > >> > > > > > > > > > > > > > > > >>> events
> > > > > > >> > > > > > > > > > > > > > > > >>> > and try to do it
> > incremental.
> > > > > > >> > > > > > > > > > > > > > > > >>> > That should improve
> > > performance.
> > > > > > >> > > > > > > > > > > > > > > > >>> > PS:
> > > > > > >> > > > > > > > > > > > > > > > >>> > I was planning to send an
> > > e-mail
> > > > > > >> tomorrow
> > > > > > >> > > > > > > announcing
> > > > > > >> > > > > > > > > > that in
> > > > > > >> > > > > > > > > > > > > > case you
> > > > > > >> > > > > > > > > > > > > > > > >>> were
> > > > > > >> > > > > > > > > > > > > > > > >>> > already working on a fix
> for
> > > > > that. I
> > > > > > >> > assume
> > > > > > >> > > > you
> > > > > > >> > > > > > are
> > > > > > >> > > > > > > > not
> > > > > > >> > > > > > > > > > and I
> > > > > > >> > > > > > > > > > > > > > would
> > > > > > >> > > > > > > > > > > > > > > > be
> > > > > > >> > > > > > > > > > > > > > > > >>> > sending a PR soon.
> > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > >> > > > > > > > > > > > > > > > >>> > On Tue, Nov 21, 2023 at
> > > 6:09 PM
> > > > > > Martin
> > > > > > >> > > Weiler
> > > > > > >> > > > > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > >> > > > > > > > > > > > > > > > >>> > wrote:
> > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > >> > > > > > > > > > > > > > > > >>> > > I looked into the new
> > > examples
> > > > > > using
> > > > > > >> > > > > data-index
> > > > > > >> > > > > > > > > > persistence
> > > > > > >> > > > > > > > > > > > > > addon -
> > > > > > >> > > > > > > > > > > > > > > > >>> Neus'
> > > > > > >> > > > > > > > > > > > > > > > >>> > > PR#1813 [1] for
> serverless
> > > and
> > > > > > >> Pere's
> > > > > > >> > > > branch
> > > > > > >> > > > > > [2]
> > > > > > >> > > > > > > > for
> > > > > > >> > > > > > > > > > > > workflow
> > > > > > >> > > > > > > > > > > > > > > > (great
> > > > > > >> > > > > > > > > > > > > > > > >>> job
> > > > > > >> > > > > > > > > > > > > > > > >>> > > both!) - and they work
> > > without
> > > > > > >> issues
> > > > > > >> > > using
> > > > > > >> > > > > > > single
> > > > > > >> > > > > > > > > > > > requests.
> > > > > > >> > > > > > > > > > > > > > > > >>> However, under
> > > > > > >> > > > > > > > > > > > > > > > >>> > > some load (I used 'ab'
> for
> > > > > testing
> > > > > > >> > with a
> > > > > > >> > > > > light
> > > > > > >> > > > > > > > > > > > concurrency of
> > > > > > >> > > > > > > > > > > > > > 10
> > > > > > >> > > > > > > > > > > > > > > > >>> parallel
> > > > > > >> > > > > > > > > > > > > > > > >>> > > requests) I ran into the
> > > > > following
> > > > > > >> > > > problems:
> > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > >> > > > > > > > > > > > > > > > >>> > > (1) Large number of
> > > > > insert/delete
> > > > > > >> calls
> > > > > > >> > > > (eg.
> > > > > > >> > > > > > for
> > > > > > >> > > > > > > > > tables
> > > > > > >> > > > > > > > > > > > such as
> > > > > > >> > > > > > > > > > > > > > > > >>> nodes,
> > > > > > >> > > > > > > > > > > > > > > > >>> > > definitions, etc)
> > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > >> > > > > > > > > > > > > > > > >>> > > (2) Hibernate
> > > > > > >> OptimisticLockExceptions
> > > > > > >> > /
> > > > > > >> > > > > > > > > > > > StaleStateExceptions
> > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > >> > > > > > > > > > > > > > > > >>> > > (3) DB deadlocks
> > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > >> > > > > > > > > > > > > > > > >>> > > (4) Error responses,
> slow
> > > > > response
> > > > > > >> > times
> > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > >> > > > > > > > > > > > > > > > >>> > > The reason I am reaching
> > out
> > > > > with
> > > > > > >> this
> > > > > > >> > > > topic
> > > > > > >> > > > > > here
> > > > > > >> > > > > > > > is
> > > > > > >> > > > > > > > > to
> > > > > > >> > > > > > > > > > > > find
> > > > > > >> > > > > > > > > > > > > > out if
> > > > > > >> > > > > > > > > > > > > > > > >>> we are
> > > > > > >> > > > > > > > > > > > > > > > >>> > > aware of this issue, and
> > if
> > > > > > someone
> > > > > > >> is
> > > > > > >> > > > > already
> > > > > > >> > > > > > > > > looking
> > > > > > >> > > > > > > > > > > > into or
> > > > > > >> > > > > > > > > > > > > > > > being
> > > > > > >> > > > > > > > > > > > > > > > >>> > > assigned to it?
> > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > >> > > > > > > > > > > > > > > > >>> > > Thanks,
> > > > > > >> > > > > > > > > > > > > > > > >>> > > Martin
> > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > >> > > > > > > > > > > > > > > > >>> > > [1]
> > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> >
> > > https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > > >> > > > > > > > > > > > > > > > >>> > > [2]
> > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > >
> > > > > > >>
> > > > ---------------------------------------------------------------------
> > > > > > >> > > > > > > > > > > > > > > > >>> > > To unsubscribe, e-mail:
> > > > > > >> > > > > > > > > dev-unsubscribe@kie.apache.org
> > > > > > >> > > > > > > > > > > > > > > > >>> > > For additional commands,
> > > > e-mail:
> > > > > > >> > > > > > > > > > dev-help@kie.apache.org
> > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > >
> > > > > > >> >
> > > > >
> ---------------------------------------------------------------------
> > > > > > >> > > > > > > > > > > > > > > > >>> To unsubscribe, e-mail:
> > > > > > >> > > > > > > dev-unsubscribe@kie.apache.org
> > > > > > >> > > > > > > > > > > > > > > > >>> For additional commands,
> > e-mail:
> > > > > > >> > > > > > > > dev-help@kie.apache.org
> > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > >
> > > > > > >>
> > > > ---------------------------------------------------------------------
> > > > > > >> > > > > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > > > > >> > > > > > dev-unsubscribe@kie.apache.org
> > > > > > >> > > > > > > > > > > > > > > > > For additional commands,
> e-mail:
> > > > > > >> > > > > > > dev-help@kie.apache.org
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > >
> > > > > > >> >
> > > > >
> ---------------------------------------------------------------------
> > > > > > >> > > > > > > > > > > > > > To unsubscribe, e-mail:
> >

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Francisco Javier Tirado Sarti <ft...@redhat.com>]

Hi Richard,
The "disabling" mechanism is based on Springboot or Quarkus configuration
(so yes, in Quarkus, the easiest way is to add a property on
application.properties, but you can use any mechanism as described here
https://quarkus.io/guides/config-reference#configuration-sources)
The properties are (all enabled by default):
kogito.events.processinstances.enabled
kogito.events.usertasks.enabled
kogito.events.processdefinitions.enabled

On Wed, Feb 21, 2024 at 12:16 PM Richard Bourner <ri...@gmail.com> wrote:

> Ok Francisco, that is interesting to know. Thanks.
>
> So disabling event publishing is a different configuration than disabling
> the data audit, correct?
> If yes, are they both done in pom.xml + application.properties?
>
> Le mer. 21 févr. 2024, 06:03, Francisco Javier Tirado Sarti <
> ftirados@redhat.com> a écrit :
>
> > Richard,
> > You can also disable event publishing (which is the mechanism to
> sincronize
> > runtimes with dataindex). In any case, since this mechanism is async, the
> > performance of the process execution won't be affected, even if enabled.
> >
> > On Wed, Feb 21, 2024 at 11:24 AM Richard Bourner <ri...@gmail.com>
> wrote:
> >
> > > Got it. Thanks Enrique.
> > >
> > >
> > > Le mer. 21 févr. 2024, 00:18, Enrique Gonzalez Martinez <
> > > egonzalez@apache.org> a écrit :
> > >
> > > > Regarding your STP concerns, they were addressed in my previous
> > comment:
> > > >
> > > > STP is a concept, a process with certain constraints: no persistence
> > and
> > > > returning the outcome in the call (sync execution with no idle
> states).
> > > It
> > > > was a requirement from a user in the past. One of the requirements
> was
> > > > leaving no trail. In v7 was easy because you could disable the audit
> in
> > > > that case. Actually we have the same way to do what we did in v7 in
> > here
> > > as
> > > > you can add/remove index just removing dependencies.
> > > >
> > > > So in fact in microservice you just need to exclude the data index
> from
> > > the
> > > > app and you wont have data index.
> > > >
> > > >
> > > >
> > > > El mié, 21 feb 2024, 1:40, Richard Bourner <ri...@gmail.com>
> > escribió:
> > > >
> > > > > +1 with Martin's email.
> > > > >
> > > > > One question though in regards to Martin's point #3 and to previous
> > > > > following statement from Francisco: "*... keeping finishing
> > > > > process instances "for a while" in DataIndex was the only way for
> > users
> > > > to
> > > > > query the result of straight through processes*"
> > > > > --> Is this the only use case where data index would be needed for
> > STP?
> > > > > I am asking because clients will already get their result in the
> JSON
> > > > > returned from the synchronous REST call, so adding an extra
> computing
> > > > time
> > > > > for data index persistence does not seem right to me in the context
> > of
> > > > > decision services that are supposed to be very fast to return
> > > (typically
> > > > > rule tasks+scripts).
> > > > > Or is it that we also want to provide some GraphQL capabilities,
> even
> > > for
> > > > > STP use cases?
> > > > >
> > > > > Also, what would "*for a while*" mean exactly?  Will it be
> > > configurable?
> > > > > Will there be a default expiration value?
> > > > >
> > > > > I am assuming this is all work in progress, and you may not have
> > > answers
> > > > to
> > > > > all my questions, no problem with that.
> > > > >
> > > > > Thanks.
> > > > >
> > > > > On Tue, Feb 20, 2024 at 5:39 PM Martin Weiler
> > <mweiler@ibm.com.invalid
> > > >
> > > > > wrote:
> > > > >
> > > > > > IMO, it is good to have this discussion around data sanity now
> > > instead
> > > > of
> > > > > > putting it off until later when data has already accumulated in
> > > > > production
> > > > > > environments.
> > > > > >
> > > > > > Based on the input here, we are dealing with three types of data:
> > > > > > 1. Runtime data - active instances only, engine cleans up the
> data
> > > > > > automatically at process instance end
> > > > > > 2. Historic log data - data created by data-audit intended for
> long
> > > > term
> > > > > > storage
> > > > > > 3. Data-index data - somehow this data falls in between the two
> > > > > > aforementioned categories, with the idea of the data being
> > "recent",
> > > > but
> > > > > > not restricted to active instances only
> > > > > >
> > > > > > We'd need purge strategies for both #2 and #3 (perhaps different
> > > ones,
> > > > or
> > > > > > with different config settings) in order to prevent unlimited
> data
> > > > > growth.
> > > > > >
> > > > > > ________________________________________
> > > > > > From: Enrique Gonzalez Martinez <eg...@apache.org>
> > > > > > Sent: Monday, February 19, 2024 7:11 AM
> > > > > > To: dev@kie.apache.org
> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > > data-index
> > > > > > persistence addon
> > > > > >
> > > > > > Hi Francisco,
> > > > > > To give you more context about this.
> > > > > >
> > > > > > STP is a concept, a process with certain constraints: no
> > persistence
> > > > and
> > > > > > returning the outcome in the call (sync execution with no idle
> > > states).
> > > > > It
> > > > > > was a requirement from a user in the past. One of the
> requirements
> > > was
> > > > > > leaving no trail. In v7 was easy because you could disable the
> > audit
> > > in
> > > > > > that case. Actually we have the same way to do what we did in v7
> in
> > > > here
> > > > > as
> > > > > > you can add/remove index just removing deps.
> > > > > >
> > > > > > We have the same outcome with different approaches and STP is
> > already
> > > > > > delivered.
> > > > > >
> > > > > > El lun, 19 feb 2024 a las 14:46, Francisco Javier Tirado Sarti (<
> > > > > > ftirados@redhat.com>) escribió:
> > > > > >
> > > > > > > Regarding STP (which is not a concept that we have in the
> code. I
> > > > mean
> > > > > > STP
> > > > > > > are processes as nonSTP are), I guess, as all processes, they
> > were
> > > > kept
> > > > > > in
> > > > > > > DataIndex once completed because users wanted (and still wants)
> > to
> > > > > check
> > > > > > > the result once the call had been performed. If we want to
> leave
> > no
> > > > > trace
> > > > > > > of them in DataIndex for some reason, we will need to make it a
> > > > > > > Runtimes concept so DataIndex can handle them in a different
> way.
> > > > > > >
> > > > > > > On Mon, Feb 19, 2024 at 2:27 PM Enrique Gonzalez Martinez <
> > > > > > > egonzalez@apache.org> wrote:
> > > > > > >
> > > > > > > > Alex:
> > > > > > > > Right now the data index is working in the same way as it did
> > in
> > > v7
> > > > > > with
> > > > > > > > the emitters. The only difference between two impl is that in
> > > here
> > > > > the
> > > > > > > > storage is pgsql instead elastic search.  You are right
> > regarding
> > > > is
> > > > > a
> > > > > > > > snapshot of the last state of the process but we did never
> > define
> > > > how
> > > > > > > long
> > > > > > > > would be alive that dats Honestly i am happy right now with
> the
> > > way
> > > > > it
> > > > > > > > works. The clean up mechanism is still tbd because we still
> > need
> > > to
> > > > > > > discuss
> > > > > > > > other stuff first.
> > > > > > > >
> > > > > > > >
> > > > > > > > Regarding stp is to leave no trail because u can get the
> > outcome
> > > > > > directly
> > > > > > > > from the call. It was defined like that in v7. So there is no
> > use
> > > > for
> > > > > > the
> > > > > > > > index or the audit.
> > > > > > > >
> > > > > > > > El lun, 19 feb 2024, 14:13, Francisco Javier Tirado Sarti <
> > > > > > > > ftirados@redhat.com> escribió:
> > > > > > > >
> > > > > > > > > Hi Alex,
> > > > > > > > > There has been some confusion about the purpose of
> DataIndex.
> > > To
> > > > be
> > > > > > > > honest
> > > > > > > > > I believe they were already sorted out, but your e-mail
> makes
> > > me
> > > > > > think
> > > > > > > > that
> > > > > > > > > is not the case ;). I let Kris to clarify that with you. My
> > > view
> > > > is
> > > > > > > that
> > > > > > > > > data-index is a way to query recently closed and active
> > > processes
> > > > > > (the
> > > > > > > > key
> > > > > > > > > here is the definition of recently, which in my opinion
> > should
> > > be
> > > > > > > > > configurable)
> > > > > > > > > But, besides that discussion and being pragmatic, keeping
> > > > finishing
> > > > > > > > process
> > > > > > > > > instances "for a while" in DataIndex was the only way for
> > users
> > > > to
> > > > > > > query
> > > > > > > > > the result of straight through processes. That's a function
> > > that
> > > > > > cannot
> > > > > > > > be
> > > > > > > > > removed right now
> > > > > > > > >
> > > > > > > > > On Mon, Feb 19, 2024 at 1:33 PM Alex Porcelli <
> > > > porcelli@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > if data index was supposed to provide snapshot view of
> the
> > > > > process
> > > > > > > > > > instance… why do we keep it after the process instance is
> > > > > finished?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Feb 19, 2024 at 7:12 AM Francisco Javier Tirado
> > > Sarti <
> > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Martin.
> > > > > > > > > > > After taking a deeper look at this, I realize that the
> > > > > behaviour
> > > > > > is
> > > > > > > > the
> > > > > > > > > > > expected one.
> > > > > > > > > > > Runtimes DB does not track the completed process
> instance
> > > > > (that's
> > > > > > > > what
> > > > > > > > > > the
> > > > > > > > > > > JDBCProcessInstances warn is telling us), but
> DataIndex,
> > as
> > > > > > > expected,
> > > > > > > > > is
> > > > > > > > > > > tracking it in processes and nodes table. And yes it
> will
> > > > grow
> > > > > > over
> > > > > > > > > time.
> > > > > > > > > > > What we need is some configurable purge mechanism for
> > > > > DataIndex,
> > > > > > so
> > > > > > > > it
> > > > > > > > > > > eventually removes older completed process instances.
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Feb 13, 2024 at 12:59 PM Francisco Javier
> Tirado
> > > > Sarti
> > > > > <
> > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Martin,
> > > > > > > > > > > > Good catch!. Looks like the skipping performed for
> > > process
> > > > > > > > instances
> > > > > > > > > is
> > > > > > > > > > > > not applied to node instances. Something we
> definitely
> > > need
> > > > > to
> > > > > > > > review
> > > > > > > > > > on
> > > > > > > > > > > > the runtimes side.
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Feb 12, 2024 at 11:59 PM Martin Weiler
> > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> On a somewhat related note, testing a simple
> workflow
> > > > (start
> > > > > > ->
> > > > > > > > > script
> > > > > > > > > > > >> node -> end), I see the following messages in the
> > logs:
> > > > > > > > > > > >> 2024-02-12 22:49:50,493 28758dde544c WARN
> > > > > > > > > > > >>
> > > [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> > > > > > > > > > > >> (executor-thread-3) Skipping create of process
> > instance
> > > > id:
> > > > > > > > > > > >> 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2
> > > > > > > > > > > >>
> > > > > > > > > > > >> So far, so good. And I'd expect to see no trace of
> > this
> > > > > > process
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > >> database if I don't have data audit enabled.
> > > > > > > > > > > >>
> > > > > > > > > > > >> However, the 'processes' table contains a row with
> > > > state=2,
> > > > > > with
> > > > > > > > > > related
> > > > > > > > > > > >> entries in the 'nodes' table. In a load test, I see
> > > these
> > > > > > tables
> > > > > > > > > grow
> > > > > > > > > > > >> significantly over time. Am I missing something to
> > have
> > > > > these
> > > > > > > > > entries
> > > > > > > > > > > >> cleaned up automatically?
> > > > > > > > > > > >>
> > > > > > > > > > > >> ________________________________________
> > > > > > > > > > > >> From: Martin Weiler <mw...@ibm.com.INVALID>
> > > > > > > > > > > >> Sent: Monday, February 12, 2024 3:40 PM
> > > > > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > > > > >> Subject: [EXTERNAL] RE: [DISCUSSION] Performance
> > issues
> > > > with
> > > > > > > > > > data-index
> > > > > > > > > > > >> persistence addon
> > > > > > > > > > > >>
> > > > > > > > > > > >> Thanks everyone for your input. Based on this
> > > discussion,
> > > > I
> > > > > > > opened
> > > > > > > > > the
> > > > > > > > > > > >> following PR:
> > > > > > > > > > > >>
> > > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1985
> > > > > > > > > > > >>
> > > > > > > > > > > >> With this change, the performance seems to be stable
> > > over
> > > > > > time:
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
> > > > > > > > > > > >>
> > > > > > > > > > > >> Martin
> > > > > > > > > > > >>
> > > > > > > > > > > >> ________________________________________
> > > > > > > > > > > >> From: Gonzalo Muñoz <gm...@apache.org>
> > > > > > > > > > > >> Sent: Friday, February 9, 2024 9:42 AM
> > > > > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > > > > >> Subject: [EXTERNAL] Re: [DISCUSSION] Performance
> > issues
> > > > with
> > > > > > > > > > data-index
> > > > > > > > > > > >> persistence addon
> > > > > > > > > > > >>
> > > > > > > > > > > >> Great work Francisco,
> > > > > > > > > > > >> Martin, take a look at this link with some related
> > tips
> > > > (in
> > > > > > case
> > > > > > > > you
> > > > > > > > > > > find
> > > > > > > > > > > >> it useful):
> > > > > > > > > > > >>
> > > > > > https://www.cybertec-postgresql.com/en/index-your-foreign-key/
> > > > > > > > > > > >>
> > > > > > > > > > > >> El vie, 9 feb 2024 a las 17:20, Francisco Javier
> > Tirado
> > > > > Sarti
> > > > > > (<
> > > > > > > > > > > >> ftirados@redhat.com>) escribió:
> > > > > > > > > > > >>
> > > > > > > > > > > >> > For the moment being, we will keep JPA till we
> > exhaust
> > > > all
> > > > > > > > > > > >> possibilities,
> > > > > > > > > > > >> > let's call switching from jpa to jdbc our hidden
> > plan
> > > B
> > > > ;)
> > > > > > > > > > > >> > I already told Martin, but in order everyone to
> > know,
> > > > just
> > > > > > > after
> > > > > > > > > > > writing
> > > > > > > > > > > >> > the previous email, I thought "what if Postgres is
> > not
> > > > > > > > > automatically
> > > > > > > > > > > >> > indexing foreign keys like mysql?" and, eureka
> > > > > > > > > > > >> > Postgres doc
> > > > > > > > > > > >> >
> > > > > > https://www.postgresql.org/docs/current/ddl-constraints.html
> > > > > > > > > > > >> > Mysql doc
> > > > > > > > > > > >> >
> > > > > > > > >
> > > > >
> https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> > > > > > > > > > > >> > These are the relevant excerpt
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > *Postgresql*
> > > > > > > > > > > >> > *A foreign key must reference columns that either
> > are
> > > a
> > > > > > > primary
> > > > > > > > > key
> > > > > > > > > > or
> > > > > > > > > > > >> form
> > > > > > > > > > > >> > a unique constraint, or are columns from a
> > non-partial
> > > > > > unique
> > > > > > > > > index.
> > > > > > > > > > > >> This
> > > > > > > > > > > >> > means that the referenced columns always have an
> > index
> > > > to
> > > > > > > allow
> > > > > > > > > > > >> efficient
> > > > > > > > > > > >> > lookups on whether a referencing row has a match.
> > > Since
> > > > a
> > > > > > > DELETE
> > > > > > > > > of
> > > > > > > > > > a
> > > > > > > > > > > >> row
> > > > > > > > > > > >> > from the referenced table or an UPDATE of a
> > referenced
> > > > > > column
> > > > > > > > will
> > > > > > > > > > > >> require
> > > > > > > > > > > >> > a scan of the referencing table for rows matching
> > the
> > > > old
> > > > > > > value,
> > > > > > > > > it
> > > > > > > > > > is
> > > > > > > > > > > >> > often a good idea to index the referencing columns
> > > too.
> > > > > > > Because
> > > > > > > > > this
> > > > > > > > > > > is
> > > > > > > > > > > >> not
> > > > > > > > > > > >> > always needed, and there are many choices
> available
> > on
> > > > how
> > > > > > to
> > > > > > > > > index,
> > > > > > > > > > > the
> > > > > > > > > > > >> > declaration of a foreign key constraint does not
> > > > > > automatically
> > > > > > > > > > create
> > > > > > > > > > > an
> > > > > > > > > > > >> > index on the referencing columns.*
> > > > > > > > > > > >> > *Mysql*
> > > > > > > > > > > >> > *MySQL requires that foreign key columns be
> indexed;
> > > if
> > > > > you
> > > > > > > > > create a
> > > > > > > > > > > >> table
> > > > > > > > > > > >> > with a foreign key constraint but no index on a
> > given
> > > > > > column,
> > > > > > > an
> > > > > > > > > > index
> > > > > > > > > > > >> is
> > > > > > > > > > > >> > created. *
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > So I asked Martin to especially create an index
> for
> > > > > > > > > > > process_instance_id
> > > > > > > > > > > >> > column on nodes table
> > > > > > > > > > > >> > I think that will fix the problem detected on the
> > > thread
> > > > > > dump.
> > > > > > > > > > > >> > The simpler process test to verify queries are
> fine
> > > > still
> > > > > > > > stands,
> > > > > > > > > > > >> though ;)
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <
> > > > > > > > tzimanyi@apache.org
> > > > > > > > > >
> > > > > > > > > > > >> wrote:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > > I always preferred pure JDBC over Hibernate
> > myself,
> > > > just
> > > > > > for
> > > > > > > > the
> > > > > > > > > > > sake
> > > > > > > > > > > >> of
> > > > > > > > > > > >> > > control of what is happening :) So I would not
> -1
> > > that
> > > > > > > myself.
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > Tibor
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado
> > > > Sarti <
> > > > > > > > > > > >> > > ftirados@redhat.com>
> > > > > > > > > > > >> > > napísal(a):
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > > Hi,
> > > > > > > > > > > >> > > > Usually I do not want to talk about work in
> > > progress
> > > > > > > because
> > > > > > > > > > > >> > preliminary
> > > > > > > > > > > >> > > > conclusions are pretty volatile but, well,
> there
> > > > are a
> > > > > > > > couple
> > > > > > > > > of
> > > > > > > > > > > >> things
> > > > > > > > > > > >> > > > that can be concluded from the really valuable
> > > > > > information
> > > > > > > > > that
> > > > > > > > > > > >> Martin
> > > > > > > > > > > >> > > > provided.
> > > > > > > > > > > >> > > > 1) In order to be able to determine if the
> > number
> > > of
> > > > > > > > > statements
> > > > > > > > > > is
> > > > > > > > > > > >> > larger
> > > > > > > > > > > >> > > > than expected, I asked Martin to test with a
> > > simpler
> > > > > > > process
> > > > > > > > > > > >> > definition.
> > > > > > > > > > > >> > > > One with just three nodes: start, script and
> > end.
> > > > The
> > > > > > > script
> > > > > > > > > one
> > > > > > > > > > > >> should
> > > > > > > > > > > >> > > > change just one variable. This way we can
> > analyze
> > > if
> > > > > the
> > > > > > > > > number
> > > > > > > > > > of
> > > > > > > > > > > >> > > queries
> > > > > > > > > > > >> > > > is the expected one. From the single log
> (audit
> > > was
> > > > > > > > activated
> > > > > > > > > > > them)
> > > > > > > > > > > >> my
> > > > > > > > > > > >> > > > conclusion is that the number of
> insert/updates
> > > over
> > > > > > > > processes
> > > > > > > > > > and
> > > > > > > > > > > >> > nodes
> > > > > > > > > > > >> > > > (there a lot over task, that I will prefer to
> > skip
> > > > for
> > > > > > > now,
> > > > > > > > > baby
> > > > > > > > > > > >> steps)
> > > > > > > > > > > >> > > is
> > > > > > > > > > > >> > > > the expected one.
> > > > > > > > > > > >> > > > 2) Analysing the thread dump, we see around 15
> > > > threads
> > > > > > > > > executing
> > > > > > > > > > > >> this
> > > > > > > > > > > >> > > line
> > > > > > > > > > > >> > > > at
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > > > > > > > > > >> > > > so its pretty clear the code to be optimized
> ;).
> > > I'm
> > > > > > > > > evaluating
> > > > > > > > > > > >> > > > possibilities within JPA/Hibernate, but I'm
> > > starting
> > > > > to
> > > > > > > > think
> > > > > > > > > > that
> > > > > > > > > > > >> it
> > > > > > > > > > > >> > > might
> > > > > > > > > > > >> > > > be better to switch to JDBC and skip
> hibernate.
> > > Our
> > > > > > lives
> > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > >> > > simpler,
> > > > > > > > > > > >> > > > especially with a schema relatively simple
> like
> > > ours
> > > > > > (that
> > > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > >> my
> > > > > > > > > > > >> > > > recommendation if I was an external
> consultant)
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <
> > > > > > > > > > tzimanyi@apache.org
> > > > > > > > > > > >
> > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > > Hi,
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > this will be a bit off-topic. However as far
> > as
> > > > > > > > > performance, I
> > > > > > > > > > > >> think
> > > > > > > > > > > >> > we
> > > > > > > > > > > >> > > > > should think about that we have string
> primary
> > > > keys
> > > > > > > > (IDs). I
> > > > > > > > > > > would
> > > > > > > > > > > >> > > expect
> > > > > > > > > > > >> > > > > the database systems are much better with
> > > indexing
> > > > > > > numeric
> > > > > > > > > > keys
> > > > > > > > > > > >> than
> > > > > > > > > > > >> > > > > strings. I remember from the past, when I
> was
> > > > > working
> > > > > > > with
> > > > > > > > > > DBs,
> > > > > > > > > > > >> that
> > > > > > > > > > > >> > > > using
> > > > > > > > > > > >> > > > > strings as keys or indexes was a discouraged
> > > > > practice.
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > Best regards,
> > > > > > > > > > > >> > > > > Tibor
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler
> > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > >
> > > > > > > > > > > >> > > > > napísal(a):
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > > I changed the test to use MongoDB [1] and
> I
> > > > don't
> > > > > > see
> > > > > > > a
> > > > > > > > > > > >> performance
> > > > > > > > > > > >> > > > > > degradation with this setup [2].
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > Please keep us posted of your findings.
> > > Thanks!
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > Martin
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > [1]
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> >
> > > > > > > > > > >
> > > > > > > >
> > > > >
> > https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > > > > > > > > >> > > > > > [2]
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > ________________________________________
> > > > > > > > > > > >> > > > > > From: Francisco Javier Tirado Sarti <
> > > > > > > > ftirados@redhat.com>
> > > > > > > > > > > >> > > > > > Sent: Wednesday, February 7, 2024 11:40 AM
> > > > > > > > > > > >> > > > > > To: dev@kie.apache.org
> > > > > > > > > > > >> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION]
> > > Performance
> > > > > > > issues
> > > > > > > > > with
> > > > > > > > > > > >> > > data-index
> > > > > > > > > > > >> > > > > > persistence addon
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > yes, it can be index degradation because
> of
> > > > size,
> > > > > > but
> > > > > > > I
> > > > > > > > > > > believe
> > > > > > > > > > > >> (I
> > > > > > > > > > > >> > > > might
> > > > > > > > > > > >> > > > > be
> > > > > > > > > > > >> > > > > > wrong) the db is too small (yet) for that.
> > > > > > > > > > > >> > > > > > But, eventually, Postgres, when the DB is
> > huge
> > > > > > enough,
> > > > > > > > > > > >> unavoidably
> > > > > > > > > > > >> > > will
> > > > > > > > > > > >> > > > > > behave like the graphic that Martin sent.
> > > > > > > > > > > >> > > > > > Since I believe we are not huge enough
> > (yet),
> > > > lets
> > > > > > > rule
> > > > > > > > > out
> > > > > > > > > > > >> another
> > > > > > > > > > > >> > > > issue
> > > > > > > > > > > >> > > > > > by analysing the sql logs (I requested
> those
> > > to
> > > > > > Martin
> > > > > > > > > > offline
> > > > > > > > > > > >> and
> > > > > > > > > > > >> > he
> > > > > > > > > > > >> > > > is
> > > > > > > > > > > >> > > > > > going to kindly collect them).
> > > > > > > > > > > >> > > > > > Also Im curious to know if Mongo behave in
> > the
> > > > > same
> > > > > > > way.
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique
> > > Gonzalez
> > > > > > > > Martinez <
> > > > > > > > > > > >> > > > > > egonzalez@apache.org> wrote:
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > > Hi Francisco,
> > > > > > > > > > > >> > > > > > > I would highly recommend to check
> indexes
> > > and
> > > > > how
> > > > > > > the
> > > > > > > > > > > updates
> > > > > > > > > > > >> > work
> > > > > > > > > > > >> > > in
> > > > > > > > > > > >> > > > > > data
> > > > > > > > > > > >> > > > > > > index to avoid full scan table and lock
> > the
> > > > full
> > > > > > > > table.
> > > > > > > > > > Some
> > > > > > > > > > > >> db
> > > > > > > > > > > >> > are
> > > > > > > > > > > >> > > > > very
> > > > > > > > > > > >> > > > > > > sensitive to that.
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > El mié, 7 feb 2024, 18:41, Francisco
> > Javier
> > > > > Tirado
> > > > > > > > > Sarti <
> > > > > > > > > > > >> > > > > > > ftirados@redhat.com> escribió:
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > > Hi Martin,
> > > > > > > > > > > >> > > > > > > > While I analyze the data, let me ask
> you
> > > if
> > > > it
> > > > > > is
> > > > > > > > > > possible
> > > > > > > > > > > >> to
> > > > > > > > > > > >> > > > perform
> > > > > > > > > > > >> > > > > > > > another check (similar in a way to
> > > disabling
> > > > > > > > > data-index
> > > > > > > > > > > like
> > > > > > > > > > > >> > you
> > > > > > > > > > > >> > > > do)
> > > > > > > > > > > >> > > > > > Can
> > > > > > > > > > > >> > > > > > > > you switch to MongoDB persistence and
> > > check
> > > > if
> > > > > > the
> > > > > > > > > same
> > > > > > > > > > > >> > > degradation
> > > > > > > > > > > >> > > > > > that
> > > > > > > > > > > >> > > > > > > is
> > > > > > > > > > > >> > > > > > > > there for postgres remains?
> > > > > > > > > > > >> > > > > > > > I do not know if this is feasible but
> > will
> > > > > > > certainly
> > > > > > > > > > > >> indicate
> > > > > > > > > > > >> > the
> > > > > > > > > > > >> > > > > > problem
> > > > > > > > > > > >> > > > > > > > is on the postgres storage layer and I
> > do
> > > > not
> > > > > > > have a
> > > > > > > > > > clear
> > > > > > > > > > > >> > > > prediction
> > > > > > > > > > > >> > > > > > of
> > > > > > > > > > > >> > > > > > > > what we will see when doing this
> switch.
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin
> > > Weiler
> > > > > > > > > > > >> > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > > > wrote:
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > > > Hi Francisco,
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > thanks for your work on this
> important
> > > > > topic!
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > I would like to share some test
> > results
> > > > > here,
> > > > > > > > which
> > > > > > > > > > > might
> > > > > > > > > > > >> > help
> > > > > > > > > > > >> > > to
> > > > > > > > > > > >> > > > > > > improve
> > > > > > > > > > > >> > > > > > > > > the codebase even further. I am
> using
> > > the
> > > > > > jmeter
> > > > > > > > > based
> > > > > > > > > > > >> test
> > > > > > > > > > > >> > > case
> > > > > > > > > > > >> > > > > from
> > > > > > > > > > > >> > > > > > > > Pere
> > > > > > > > > > > >> > > > > > > > > and Enrique (thanks guys!) [1] which
> > > uses
> > > > a
> > > > > > load
> > > > > > > > of
> > > > > > > > > 30
> > > > > > > > > > > >> > threads
> > > > > > > > > > > >> > > to
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > 1) start a new process instance
> (POST)
> > > > > > > > > > > >> > > > > > > > > 2) retrieve tasks for a user (GET)
> > > > > > > > > > > >> > > > > > > > > 3) fetches task details (GET)
> > > > > > > > > > > >> > > > > > > > > 4) complete a task (POST)
> > > > > > > > > > > >> > > > > > > > > 5) execute a query on data-audit
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > With this test setup, I noticed that
> > the
> > > > > > > > performance
> > > > > > > > > > for
> > > > > > > > > > > >> the
> > > > > > > > > > > >> > > POST
> > > > > > > > > > > >> > > > > > > > > requests, in particular the one to
> > > start a
> > > > > new
> > > > > > > > > process
> > > > > > > > > > > >> > > instance,
> > > > > > > > > > > >> > > > > > > degrades
> > > > > > > > > > > >> > > > > > > > > over time - see graph [2]. If I run
> > the
> > > > same
> > > > > > > test
> > > > > > > > > > > without
> > > > > > > > > > > >> > > > > data-index,
> > > > > > > > > > > >> > > > > > > > then
> > > > > > > > > > > >> > > > > > > > > there is no such performance
> > degradation
> > > > > [3].
> > > > > > > You
> > > > > > > > > can
> > > > > > > > > > > >> find a
> > > > > > > > > > > >> > > > thread
> > > > > > > > > > > >> > > > > > > dump
> > > > > > > > > > > >> > > > > > > > > captured a few minutes into the
> first
> > > test
> > > > > > here
> > > > > > > > [4]
> > > > > > > > > > that
> > > > > > > > > > > >> > might
> > > > > > > > > > > >> > > > help
> > > > > > > > > > > >> > > > > > to
> > > > > > > > > > > >> > > > > > > > see
> > > > > > > > > > > >> > > > > > > > > some of the contention points.
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > I'd appreciate if you could take a
> > look
> > > > and
> > > > > > see
> > > > > > > if
> > > > > > > > > > there
> > > > > > > > > > > >> is
> > > > > > > > > > > >> > > > > something
> > > > > > > > > > > >> > > > > > > > that
> > > > > > > > > > > >> > > > > > > > > can be further improved based on
> your
> > > > > previous
> > > > > > > > work.
> > > > > > > > > > If
> > > > > > > > > > > >> you
> > > > > > > > > > > >> > > need
> > > > > > > > > > > >> > > > > any
> > > > > > > > > > > >> > > > > > > > > additional data, let me know, but
> > > > otherwise
> > > > > it
> > > > > > > is
> > > > > > > > > > > >> > > straightforward
> > > > > > > > > > > >> > > > > to
> > > > > > > > > > > >> > > > > > > run
> > > > > > > > > > > >> > > > > > > > > the jmeter test as well.
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > Thanks,
> > > > > > > > > > > >> > > > > > > > > Martin
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > [1]
> > > > > > > > > > > >>
> > https://github.com/pefernan/job-service-refactor-test/
> > > > > > > > > > > >> > > > > > > > > [2]
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > > > > > > > >> > > > > > > > > [3]
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > > > > > > > >> > > > > > > > > [4]
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > >
> > ________________________________________
> > > > > > > > > > > >> > > > > > > > > From: Francisco Javier Tirado Sarti
> <
> > > > > > > > > > > ftirados@redhat.com>
> > > > > > > > > > > >> > > > > > > > > Sent: Wednesday, January 17, 2024
> 9:13
> > > AM
> > > > > > > > > > > >> > > > > > > > > To: dev@kie.apache.org
> > > > > > > > > > > >> > > > > > > > > Cc: Pere Fernandez Perez
> > > > > > > > > > > >> > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION]
> > > > > > Performance
> > > > > > > > > > issues
> > > > > > > > > > > >> with
> > > > > > > > > > > >> > > > > > data-index
> > > > > > > > > > > >> > > > > > > > > persistence addon
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > Hi Alex,
> > > > > > > > > > > >> > > > > > > > > I did not take times (which depends
> > on a
> > > > > > number
> > > > > > > of
> > > > > > > > > > > >> variables
> > > > > > > > > > > >> > > that
> > > > > > > > > > > >> > > > > > > > > drastically change between
> > > environments),
> > > > > but
> > > > > > > > verify
> > > > > > > > > > > that
> > > > > > > > > > > >> the
> > > > > > > > > > > >> > > > > number
> > > > > > > > > > > >> > > > > > of
> > > > > > > > > > > >> > > > > > > > > updates has been reduced drastically
> > > > without
> > > > > > > > losing
> > > > > > > > > > > >> > > > functionality,
> > > > > > > > > > > >> > > > > > > which
> > > > > > > > > > > >> > > > > > > > is
> > > > > > > > > > > >> > > > > > > > > objectively a good thing. If before
> > the
> > > > > > change,
> > > > > > > > for
> > > > > > > > > > > every
> > > > > > > > > > > >> > node
> > > > > > > > > > > >> > > > > > > executed,
> > > > > > > > > > > >> > > > > > > > we
> > > > > > > > > > > >> > > > > > > > > have an update for every node
> > previously
> > > > > > > executed,
> > > > > > > > > so
> > > > > > > > > > > if a
> > > > > > > > > > > >> > > > process
> > > > > > > > > > > >> > > > > > have
> > > > > > > > > > > >> > > > > > > > 50
> > > > > > > > > > > >> > > > > > > > > nodes to execute, we were performing
> > > > nearly
> > > > > > > > 50*51/2
> > > > > > > > > > > >> updates,
> > > > > > > > > > > >> > > > which
> > > > > > > > > > > >> > > > > > > gives
> > > > > > > > > > > >> > > > > > > > us
> > > > > > > > > > > >> > > > > > > > > a total of  1275 updates, now we
> have
> > > just
> > > > > one
> > > > > > > for
> > > > > > > > > > every
> > > > > > > > > > > >> node
> > > > > > > > > > > >> > > > being
> > > > > > > > > > > >> > > > > > > > > executed, implying a total of 50
> > > updates.
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex
> > > > > Porcelli
> > > > > > <
> > > > > > > > > > > >> > > alex@porcelli.me>
> > > > > > > > > > > >> > > > > > > wrote:
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > Francisco,
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > I noticed that your PR has been
> > > merged,
> > > > > but
> > > > > > I
> > > > > > > > was
> > > > > > > > > > > >> expecting
> > > > > > > > > > > >> > > (at
> > > > > > > > > > > >> > > > > > least
> > > > > > > > > > > >> > > > > > > > > > was my understanding from this
> > thread)
> > > > > that
> > > > > > > > before
> > > > > > > > > > > >> merging
> > > > > > > > > > > >> > > some
> > > > > > > > > > > >> > > > > > > > > > benchmark data would be shared in
> > > > advance
> > > > > -
> > > > > > to
> > > > > > > > > > assess
> > > > > > > > > > > >> the
> > > > > > > > > > > >> > > > > > > cost/benefit
> > > > > > > > > > > >> > > > > > > > > > of such a decent size change.
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > Do you have any information to
> > share?
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM
> > > > Francisco
> > > > > > > Javier
> > > > > > > > > > > Tirado
> > > > > > > > > > > >> > Sarti
> > > > > > > > > > > >> > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > Yes, as intended, now we have
> one
> > > > select
> > > > > > and
> > > > > > > > one
> > > > > > > > > > > >> > > > insert/update
> > > > > > > > > > > >> > > > > > per
> > > > > > > > > > > >> > > > > > > > node
> > > > > > > > > > > >> > > > > > > > > > > event.
> > > > > > > > > > > >> > > > > > > > > > > I moved the PR as ready for
> review
> > > and
> > > > > > give
> > > > > > > > > @Pere
> > > > > > > > > > > >> > Fernandez
> > > > > > > > > > > >> > > > > Perez
> > > > > > > > > > > >> > > > > > > > > > > <pe...@redhat.com>
> permission
> > to
> > > > the
> > > > > > > > branch
> > > > > > > > > so
> > > > > > > > > > > he
> > > > > > > > > > > >> can
> > > > > > > > > > > >> > > > edit
> > > > > > > > > > > >> > > > > it
> > > > > > > > > > > >> > > > > > > in
> > > > > > > > > > > >> > > > > > > > > the
> > > > > > > > > > > >> > > > > > > > > > > next two weeks (Ill be on PTO)
> if
> > > > > > desired,
> > > > > > > > > before
> > > > > > > > > > > >> > merging.
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM
> > Alex
> > > > > > > Porcelli
> > > > > > > > <
> > > > > > > > > > > >> > > > > alex@porcelli.me>
> > > > > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > Cool, thank you Francisco!
> > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > Did you manage to get some
> > > > preliminary
> > > > > > > data
> > > > > > > > > > about
> > > > > > > > > > > >> > > > > improvements?
> > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > On Thu, Dec 21, 2023 at
> 11:52 AM
> > > > > > Francisco
> > > > > > > > > > Javier
> > > > > > > > > > > >> > Tirado
> > > > > > > > > > > >> > > > > Sarti
> > > > > > > > > > > >> > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > Yes, after some delay
> because
> > of
> > > > > > > quarkus 3
> > > > > > > > > > > >> migration.
> > > > > > > > > > > >> > > Im
> > > > > > > > > > > >> > > > > > > refining
> > > > > > > > > > > >> > > > > > > > > > this
> > > > > > > > > > > >> > > > > > > > > > > > > draft PR
> > > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > >
> > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > On Thu, Dec 21, 2023 at
> > 5:48 PM
> > > > Alex
> > > > > > > > > Porcelli
> > > > > > > > > > <
> > > > > > > > > > > >> > > > > > > alex@porcelli.me>
> > > > > > > > > > > >> > > > > > > > > > wrote:
> > > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > Any update or new findings
> > on
> > > > this
> > > > > > > > topic?
> > > > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > On Tue, Nov 28, 2023 at
> > > 8:38 AM
> > > > > > > > Francisco
> > > > > > > > > > > Javier
> > > > > > > > > > > >> > > Tirado
> > > > > > > > > > > >> > > > > > Sarti
> > > > > > > > > > > >> > > > > > > > > > > > > > <ft...@redhat.com>
> > wrote:
> > > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > >> > > > > > > > > > > > > > > After considering
> > different
> > > > > > options
> > > > > > > to
> > > > > > > > > > > improve
> > > > > > > > > > > >> > > > > > performance,
> > > > > > > > > > > >> > > > > > > > we
> > > > > > > > > > > >> > > > > > > > > > feel
> > > > > > > > > > > >> > > > > > > > > > > > that
> > > > > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > > > > > >> > > > > > > > > > > > > > > is time to "partially"
> > move
> > > > away
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > > > >> current
> > > > > > > > > > > >> > > Map
> > > > > > > > > > > >> > > > > > style
> > > > > > > > > > > >> > > > > > > > > > > > interface (
> > > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > > > > > > >> > > > > > > > > > > > > > )
> > > > > > > > > > > >> > > > > > > > > > > > > > > which was shared with
> > > Trusty,
> > > > to
> > > > > > one
> > > > > > > > > more
> > > > > > > > > > > >> > suitable
> > > > > > > > > > > >> > > > for
> > > > > > > > > > > >> > > > > > > usage
> > > > > > > > > > > >> > > > > > > > > > with a
> > > > > > > > > > > >> > > > > > > > > > > > > > > relational DB like
> > > postgresql
> > > > > (but
> > > > > > > > still
> > > > > > > > > > > >> > compatible
> > > > > > > > > > > >> > > > > with
> > > > > > > > > > > >> > > > > > > big
> > > > > > > > > > > >> > > > > > > > > > table
> > > > > > > > > > > >> > > > > > > > > > > > dbs).
> > > > > > > > > > > >> > > > > > > > > > > > > > > The idea will be to
> > replace
> > > > > > generic
> > > > > > > > > > Storage
> > > > > > > > > > > >> > > interface
> > > > > > > > > > > >> > > > > by
> > > > > > > > > > > >> > > > > > > four
> > > > > > > > > > > >> > > > > > > > > > > > specific
> > > > > > > > > > > >> > > > > > > > > > > > > > > interfaces (which will
> > > inherit
> > > > > > from
> > > > > > > a
> > > > > > > > > > common
> > > > > > > > > > > >> one
> > > > > > > > > > > >> > > that
> > > > > > > > > > > >> > > > > > keeps
> > > > > > > > > > > >> > > > > > > > the
> > > > > > > > > > > >> > > > > > > > > > query
> > > > > > > > > > > >> > > > > > > > > > > > > > part
> > > > > > > > > > > >> > > > > > > > > > > > > > > at is it. with get and
> > query
> > > > > > > methods),
> > > > > > > > > > that
> > > > > > > > > > > >> will
> > > > > > > > > > > >> > > > > include
> > > > > > > > > > > >> > > > > > > the
> > > > > > > > > > > >> > > > > > > > > > required
> > > > > > > > > > > >> > > > > > > > > > > > > > > modification operations
> > for
> > > > the
> > > > > > four
> > > > > > > > > > > DataIndex
> > > > > > > > > > > >> > > > > "domains":
> > > > > > > > > > > >> > > > > > > > > > > > > > processinstance,
> > > > > > > > > > > >> > > > > > > > > > > > > > > usertask,
> > processdefinitions
> > > > and
> > > > > > > jobs.
> > > > > > > > > > Those
> > > > > > > > > > > >> > > > interfaces
> > > > > > > > > > > >> > > > > > > will
> > > > > > > > > > > >> > > > > > > > > > define
> > > > > > > > > > > >> > > > > > > > > > > > > > methods
> > > > > > > > > > > >> > > > > > > > > > > > > > > like addNode,
> addVariable,
> > > > > > > updateTask,
> > > > > > > > > > > >> > > > > addAttachment.....
> > > > > > > > > > > >> > > > > > > > that
> > > > > > > > > > > >> > > > > > > > > > will
> > > > > > > > > > > >> > > > > > > > > > > > allow
> > > > > > > > > > > >> > > > > > > > > > > > > > > the persistent layer
> > > > > > implementation
> > > > > > > > to
> > > > > > > > > > just
> > > > > > > > > > > >> > update
> > > > > > > > > > > >> > > > the
> > > > > > > > > > > >> > > > > > > > needed
> > > > > > > > > > > >> > > > > > > > > > info
> > > > > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > > > >> > > > > > > > > > > > > > > DB  (for example, for
> > > addNode
> > > > in
> > > > > > > > > Postgres,
> > > > > > > > > > > >> just
> > > > > > > > > > > >> > > > insert
> > > > > > > > > > > >> > > > > a
> > > > > > > > > > > >> > > > > > > row
> > > > > > > > > > > >> > > > > > > > > into
> > > > > > > > > > > >> > > > > > > > > > > > nodes
> > > > > > > > > > > >> > > > > > > > > > > > > > > table, for addNode in
> > Mongo,
> > > > > > > basically
> > > > > > > > > the
> > > > > > > > > > > >> same
> > > > > > > > > > > >> > > > atomic
> > > > > > > > > > > >> > > > > > > upsert
> > > > > > > > > > > >> > > > > > > > > > > > operation
> > > > > > > > > > > >> > > > > > > > > > > > > > > that is currently done).
> > > > > > Therefore,
> > > > > > > we
> > > > > > > > > > > >> increase
> > > > > > > > > > > >> > > > > > performance
> > > > > > > > > > > >> > > > > > > > for
> > > > > > > > > > > >> > > > > > > > > > > > Postgres
> > > > > > > > > > > >> > > > > > > > > > > > > > > and keep the current one
> > for
> > > > > > Mongo.
> > > > > > > > The
> > > > > > > > > > > >> current
> > > > > > > > > > > >> > DB
> > > > > > > > > > > >> > > > > > schemas
> > > > > > > > > > > >> > > > > > > > > won't
> > > > > > > > > > > >> > > > > > > > > > be
> > > > > > > > > > > >> > > > > > > > > > > > > > > touched.
> > > > > > > > > > > >> > > > > > > > > > > > > > > Since the code change is
> > > > large,
> > > > > I
> > > > > > do
> > > > > > > > not
> > > > > > > > > > > think
> > > > > > > > > > > >> > I'll
> > > > > > > > > > > >> > > > be
> > > > > > > > > > > >> > > > > > able
> > > > > > > > > > > >> > > > > > > > to
> > > > > > > > > > > >> > > > > > > > > > have
> > > > > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > > > >> > > > > > > > > > > > > > PR
> > > > > > > > > > > >> > > > > > > > > > > > > > > ready till next week.
> > > > > > > > > > > >> > > > > > > > > > > > > > > But before starting,
> > please
> > > > let
> > > > > me
> > > > > > > > know
> > > > > > > > > if
> > > > > > > > > > > >> that
> > > > > > > > > > > >> > > > > approach
> > > > > > > > > > > >> > > > > > is
> > > > > > > > > > > >> > > > > > > > > fine
> > > > > > > > > > > >> > > > > > > > > > for
> > > > > > > > > > > >> > > > > > > > > > > > you.
> > > > > > > > > > > >> > > > > > > > > > > > > > > Best regards.
> > > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> > > > 6:55 PM
> > > > > > Alex
> > > > > > > > > > > Porcelli
> > > > > > > > > > > >> <
> > > > > > > > > > > >> > > > > > > > > alex@porcelli.me>
> > > > > > > > > > > >> > > > > > > > > > > > wrote:
> > > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > > > Thank you Francisco to
> > > > getting
> > > > > > > > deeper
> > > > > > > > > on
> > > > > > > > > > > >> this…
> > > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > > > Looking forward to see
> > the
> > > > > > results
> > > > > > > > of
> > > > > > > > > > your
> > > > > > > > > > > >> > > > suggested
> > > > > > > > > > > >> > > > > > > > > > improvements.
> > > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023
> at
> > > > > 9:40 AM
> > > > > > > > > > Francisco
> > > > > > > > > > > >> > Javier
> > > > > > > > > > > >> > > > > Tirado
> > > > > > > > > > > >> > > > > > > > > Sarti <
> > > > > > > > > > > >> > > > > > > > > > > > > > > > ftirados@redhat.com>
> > > wrote:
> > > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > > > > I forgot to attach
> the
> > > > > queries
> > > > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > > > > On Fri, Nov 24, 2023
> > at
> > > > > > 3:04 PM
> > > > > > > > > > > Francisco
> > > > > > > > > > > >> > > Javier
> > > > > > > > > > > >> > > > > > Tirado
> > > > > > > > > > > >> > > > > > > > > > Sarti <
> > > > > > > > > > > >> > > > > > > > > > > > > > > > > ftirados@redhat.com
> >
> > > > wrote:
> > > > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > > > >> Hi,
> > > > > > > > > > > >> > > > > > > > > > > > > > > > >> A brief update on
> > this
> > > > > topic.
> > > > > > > > > > > >> > > > > > > > > > > > > > > > >> After doing a
> simple
> > > test
> > > > > > with
> > > > > > > > > > example
> > > > > > > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > > > > > >> > > > > > > > > > > > > > > > ,
> > > > > > > > > > > >> > > > > > > > > > > > > > > > >> the number of
> updates
> > > > over
> > > > > > > Nodes
> > > > > > > > > > table
> > > > > > > > > > > is
> > > > > > > > > > > >> > n*n,
> > > > > > > > > > > >> > > > so
> > > > > > > > > > > >> > > > > we
> > > > > > > > > > > >> > > > > > > > > manage
> > > > > > > > > > > >> > > > > > > > > > to
> > > > > > > > > > > >> > > > > > > > > > > > > > obtain a
> > > > > > > > > > > >> > > > > > > > > > > > > > > > >> perfect quadratic
> > > > > performance
> > > > > > > > > > > >> degradation.
> > > > > > > > > > > >> > The
> > > > > > > > > > > >> > > > > > problem
> > > > > > > > > > > >> > > > > > > > is
> > > > > > > > > > > >> > > > > > > > > > worse
> > > > > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > > > >> > > > > > > > > > > > > > > > case
> > > > > > > > > > > >> > > > > > > > > > > > > > > > >> of Serverless
> > Workflow
> > > > than
> > > > > > in
> > > > > > > > BPMN
> > > > > > > > > > > >> because
> > > > > > > > > > > >> > we
> > > > > > > > > > > >> > > > the
> > > > > > > > > > > >> > > > > > > > number
> > > > > > > > > > > >> > > > > > > >
>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Richard Bourner <ri...@gmail.com>]

Ok Francisco, that is interesting to know. Thanks.

So disabling event publishing is a different configuration than disabling
the data audit, correct?
If yes, are they both done in pom.xml + application.properties?

Le mer. 21 févr. 2024, 06:03, Francisco Javier Tirado Sarti <
ftirados@redhat.com> a écrit :

> Richard,
> You can also disable event publishing (which is the mechanism to sincronize
> runtimes with dataindex). In any case, since this mechanism is async, the
> performance of the process execution won't be affected, even if enabled.
>
> On Wed, Feb 21, 2024 at 11:24 AM Richard Bourner <ri...@gmail.com> wrote:
>
> > Got it. Thanks Enrique.
> >
> >
> > Le mer. 21 févr. 2024, 00:18, Enrique Gonzalez Martinez <
> > egonzalez@apache.org> a écrit :
> >
> > > Regarding your STP concerns, they were addressed in my previous
> comment:
> > >
> > > STP is a concept, a process with certain constraints: no persistence
> and
> > > returning the outcome in the call (sync execution with no idle states).
> > It
> > > was a requirement from a user in the past. One of the requirements was
> > > leaving no trail. In v7 was easy because you could disable the audit in
> > > that case. Actually we have the same way to do what we did in v7 in
> here
> > as
> > > you can add/remove index just removing dependencies.
> > >
> > > So in fact in microservice you just need to exclude the data index from
> > the
> > > app and you wont have data index.
> > >
> > >
> > >
> > > El mié, 21 feb 2024, 1:40, Richard Bourner <ri...@gmail.com>
> escribió:
> > >
> > > > +1 with Martin's email.
> > > >
> > > > One question though in regards to Martin's point #3 and to previous
> > > > following statement from Francisco: "*... keeping finishing
> > > > process instances "for a while" in DataIndex was the only way for
> users
> > > to
> > > > query the result of straight through processes*"
> > > > --> Is this the only use case where data index would be needed for
> STP?
> > > > I am asking because clients will already get their result in the JSON
> > > > returned from the synchronous REST call, so adding an extra computing
> > > time
> > > > for data index persistence does not seem right to me in the context
> of
> > > > decision services that are supposed to be very fast to return
> > (typically
> > > > rule tasks+scripts).
> > > > Or is it that we also want to provide some GraphQL capabilities, even
> > for
> > > > STP use cases?
> > > >
> > > > Also, what would "*for a while*" mean exactly?  Will it be
> > configurable?
> > > > Will there be a default expiration value?
> > > >
> > > > I am assuming this is all work in progress, and you may not have
> > answers
> > > to
> > > > all my questions, no problem with that.
> > > >
> > > > Thanks.
> > > >
> > > > On Tue, Feb 20, 2024 at 5:39 PM Martin Weiler
> <mweiler@ibm.com.invalid
> > >
> > > > wrote:
> > > >
> > > > > IMO, it is good to have this discussion around data sanity now
> > instead
> > > of
> > > > > putting it off until later when data has already accumulated in
> > > > production
> > > > > environments.
> > > > >
> > > > > Based on the input here, we are dealing with three types of data:
> > > > > 1. Runtime data - active instances only, engine cleans up the data
> > > > > automatically at process instance end
> > > > > 2. Historic log data - data created by data-audit intended for long
> > > term
> > > > > storage
> > > > > 3. Data-index data - somehow this data falls in between the two
> > > > > aforementioned categories, with the idea of the data being
> "recent",
> > > but
> > > > > not restricted to active instances only
> > > > >
> > > > > We'd need purge strategies for both #2 and #3 (perhaps different
> > ones,
> > > or
> > > > > with different config settings) in order to prevent unlimited data
> > > > growth.
> > > > >
> > > > > ________________________________________
> > > > > From: Enrique Gonzalez Martinez <eg...@apache.org>
> > > > > Sent: Monday, February 19, 2024 7:11 AM
> > > > > To: dev@kie.apache.org
> > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > data-index
> > > > > persistence addon
> > > > >
> > > > > Hi Francisco,
> > > > > To give you more context about this.
> > > > >
> > > > > STP is a concept, a process with certain constraints: no
> persistence
> > > and
> > > > > returning the outcome in the call (sync execution with no idle
> > states).
> > > > It
> > > > > was a requirement from a user in the past. One of the requirements
> > was
> > > > > leaving no trail. In v7 was easy because you could disable the
> audit
> > in
> > > > > that case. Actually we have the same way to do what we did in v7 in
> > > here
> > > > as
> > > > > you can add/remove index just removing deps.
> > > > >
> > > > > We have the same outcome with different approaches and STP is
> already
> > > > > delivered.
> > > > >
> > > > > El lun, 19 feb 2024 a las 14:46, Francisco Javier Tirado Sarti (<
> > > > > ftirados@redhat.com>) escribió:
> > > > >
> > > > > > Regarding STP (which is not a concept that we have in the code. I
> > > mean
> > > > > STP
> > > > > > are processes as nonSTP are), I guess, as all processes, they
> were
> > > kept
> > > > > in
> > > > > > DataIndex once completed because users wanted (and still wants)
> to
> > > > check
> > > > > > the result once the call had been performed. If we want to leave
> no
> > > > trace
> > > > > > of them in DataIndex for some reason, we will need to make it a
> > > > > > Runtimes concept so DataIndex can handle them in a different way.
> > > > > >
> > > > > > On Mon, Feb 19, 2024 at 2:27 PM Enrique Gonzalez Martinez <
> > > > > > egonzalez@apache.org> wrote:
> > > > > >
> > > > > > > Alex:
> > > > > > > Right now the data index is working in the same way as it did
> in
> > v7
> > > > > with
> > > > > > > the emitters. The only difference between two impl is that in
> > here
> > > > the
> > > > > > > storage is pgsql instead elastic search.  You are right
> regarding
> > > is
> > > > a
> > > > > > > snapshot of the last state of the process but we did never
> define
> > > how
> > > > > > long
> > > > > > > would be alive that dats Honestly i am happy right now with the
> > way
> > > > it
> > > > > > > works. The clean up mechanism is still tbd because we still
> need
> > to
> > > > > > discuss
> > > > > > > other stuff first.
> > > > > > >
> > > > > > >
> > > > > > > Regarding stp is to leave no trail because u can get the
> outcome
> > > > > directly
> > > > > > > from the call. It was defined like that in v7. So there is no
> use
> > > for
> > > > > the
> > > > > > > index or the audit.
> > > > > > >
> > > > > > > El lun, 19 feb 2024, 14:13, Francisco Javier Tirado Sarti <
> > > > > > > ftirados@redhat.com> escribió:
> > > > > > >
> > > > > > > > Hi Alex,
> > > > > > > > There has been some confusion about the purpose of DataIndex.
> > To
> > > be
> > > > > > > honest
> > > > > > > > I believe they were already sorted out, but your e-mail makes
> > me
> > > > > think
> > > > > > > that
> > > > > > > > is not the case ;). I let Kris to clarify that with you. My
> > view
> > > is
> > > > > > that
> > > > > > > > data-index is a way to query recently closed and active
> > processes
> > > > > (the
> > > > > > > key
> > > > > > > > here is the definition of recently, which in my opinion
> should
> > be
> > > > > > > > configurable)
> > > > > > > > But, besides that discussion and being pragmatic, keeping
> > > finishing
> > > > > > > process
> > > > > > > > instances "for a while" in DataIndex was the only way for
> users
> > > to
> > > > > > query
> > > > > > > > the result of straight through processes. That's a function
> > that
> > > > > cannot
> > > > > > > be
> > > > > > > > removed right now
> > > > > > > >
> > > > > > > > On Mon, Feb 19, 2024 at 1:33 PM Alex Porcelli <
> > > porcelli@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > if data index was supposed to provide snapshot view of the
> > > > process
> > > > > > > > > instance… why do we keep it after the process instance is
> > > > finished?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Feb 19, 2024 at 7:12 AM Francisco Javier Tirado
> > Sarti <
> > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > >
> > > > > > > > > > Hi Martin.
> > > > > > > > > > After taking a deeper look at this, I realize that the
> > > > behaviour
> > > > > is
> > > > > > > the
> > > > > > > > > > expected one.
> > > > > > > > > > Runtimes DB does not track the completed process instance
> > > > (that's
> > > > > > > what
> > > > > > > > > the
> > > > > > > > > > JDBCProcessInstances warn is telling us), but DataIndex,
> as
> > > > > > expected,
> > > > > > > > is
> > > > > > > > > > tracking it in processes and nodes table. And yes it will
> > > grow
> > > > > over
> > > > > > > > time.
> > > > > > > > > > What we need is some configurable purge mechanism for
> > > > DataIndex,
> > > > > so
> > > > > > > it
> > > > > > > > > > eventually removes older completed process instances.
> > > > > > > > > >
> > > > > > > > > > On Tue, Feb 13, 2024 at 12:59 PM Francisco Javier Tirado
> > > Sarti
> > > > <
> > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Martin,
> > > > > > > > > > > Good catch!. Looks like the skipping performed for
> > process
> > > > > > > instances
> > > > > > > > is
> > > > > > > > > > > not applied to node instances. Something we definitely
> > need
> > > > to
> > > > > > > review
> > > > > > > > > on
> > > > > > > > > > > the runtimes side.
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Feb 12, 2024 at 11:59 PM Martin Weiler
> > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> On a somewhat related note, testing a simple workflow
> > > (start
> > > > > ->
> > > > > > > > script
> > > > > > > > > > >> node -> end), I see the following messages in the
> logs:
> > > > > > > > > > >> 2024-02-12 22:49:50,493 28758dde544c WARN
> > > > > > > > > > >>
> > [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> > > > > > > > > > >> (executor-thread-3) Skipping create of process
> instance
> > > id:
> > > > > > > > > > >> 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2
> > > > > > > > > > >>
> > > > > > > > > > >> So far, so good. And I'd expect to see no trace of
> this
> > > > > process
> > > > > > in
> > > > > > > > the
> > > > > > > > > > >> database if I don't have data audit enabled.
> > > > > > > > > > >>
> > > > > > > > > > >> However, the 'processes' table contains a row with
> > > state=2,
> > > > > with
> > > > > > > > > related
> > > > > > > > > > >> entries in the 'nodes' table. In a load test, I see
> > these
> > > > > tables
> > > > > > > > grow
> > > > > > > > > > >> significantly over time. Am I missing something to
> have
> > > > these
> > > > > > > > entries
> > > > > > > > > > >> cleaned up automatically?
> > > > > > > > > > >>
> > > > > > > > > > >> ________________________________________
> > > > > > > > > > >> From: Martin Weiler <mw...@ibm.com.INVALID>
> > > > > > > > > > >> Sent: Monday, February 12, 2024 3:40 PM
> > > > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > > > >> Subject: [EXTERNAL] RE: [DISCUSSION] Performance
> issues
> > > with
> > > > > > > > > data-index
> > > > > > > > > > >> persistence addon
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks everyone for your input. Based on this
> > discussion,
> > > I
> > > > > > opened
> > > > > > > > the
> > > > > > > > > > >> following PR:
> > > > > > > > > > >>
> > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1985
> > > > > > > > > > >>
> > > > > > > > > > >> With this change, the performance seems to be stable
> > over
> > > > > time:
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
> > > > > > > > > > >>
> > > > > > > > > > >> Martin
> > > > > > > > > > >>
> > > > > > > > > > >> ________________________________________
> > > > > > > > > > >> From: Gonzalo Muñoz <gm...@apache.org>
> > > > > > > > > > >> Sent: Friday, February 9, 2024 9:42 AM
> > > > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > > > >> Subject: [EXTERNAL] Re: [DISCUSSION] Performance
> issues
> > > with
> > > > > > > > > data-index
> > > > > > > > > > >> persistence addon
> > > > > > > > > > >>
> > > > > > > > > > >> Great work Francisco,
> > > > > > > > > > >> Martin, take a look at this link with some related
> tips
> > > (in
> > > > > case
> > > > > > > you
> > > > > > > > > > find
> > > > > > > > > > >> it useful):
> > > > > > > > > > >>
> > > > > https://www.cybertec-postgresql.com/en/index-your-foreign-key/
> > > > > > > > > > >>
> > > > > > > > > > >> El vie, 9 feb 2024 a las 17:20, Francisco Javier
> Tirado
> > > > Sarti
> > > > > (<
> > > > > > > > > > >> ftirados@redhat.com>) escribió:
> > > > > > > > > > >>
> > > > > > > > > > >> > For the moment being, we will keep JPA till we
> exhaust
> > > all
> > > > > > > > > > >> possibilities,
> > > > > > > > > > >> > let's call switching from jpa to jdbc our hidden
> plan
> > B
> > > ;)
> > > > > > > > > > >> > I already told Martin, but in order everyone to
> know,
> > > just
> > > > > > after
> > > > > > > > > > writing
> > > > > > > > > > >> > the previous email, I thought "what if Postgres is
> not
> > > > > > > > automatically
> > > > > > > > > > >> > indexing foreign keys like mysql?" and, eureka
> > > > > > > > > > >> > Postgres doc
> > > > > > > > > > >> >
> > > > > https://www.postgresql.org/docs/current/ddl-constraints.html
> > > > > > > > > > >> > Mysql doc
> > > > > > > > > > >> >
> > > > > > > >
> > > > https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> > > > > > > > > > >> > These are the relevant excerpt
> > > > > > > > > > >> >
> > > > > > > > > > >> > *Postgresql*
> > > > > > > > > > >> > *A foreign key must reference columns that either
> are
> > a
> > > > > > primary
> > > > > > > > key
> > > > > > > > > or
> > > > > > > > > > >> form
> > > > > > > > > > >> > a unique constraint, or are columns from a
> non-partial
> > > > > unique
> > > > > > > > index.
> > > > > > > > > > >> This
> > > > > > > > > > >> > means that the referenced columns always have an
> index
> > > to
> > > > > > allow
> > > > > > > > > > >> efficient
> > > > > > > > > > >> > lookups on whether a referencing row has a match.
> > Since
> > > a
> > > > > > DELETE
> > > > > > > > of
> > > > > > > > > a
> > > > > > > > > > >> row
> > > > > > > > > > >> > from the referenced table or an UPDATE of a
> referenced
> > > > > column
> > > > > > > will
> > > > > > > > > > >> require
> > > > > > > > > > >> > a scan of the referencing table for rows matching
> the
> > > old
> > > > > > value,
> > > > > > > > it
> > > > > > > > > is
> > > > > > > > > > >> > often a good idea to index the referencing columns
> > too.
> > > > > > Because
> > > > > > > > this
> > > > > > > > > > is
> > > > > > > > > > >> not
> > > > > > > > > > >> > always needed, and there are many choices available
> on
> > > how
> > > > > to
> > > > > > > > index,
> > > > > > > > > > the
> > > > > > > > > > >> > declaration of a foreign key constraint does not
> > > > > automatically
> > > > > > > > > create
> > > > > > > > > > an
> > > > > > > > > > >> > index on the referencing columns.*
> > > > > > > > > > >> > *Mysql*
> > > > > > > > > > >> > *MySQL requires that foreign key columns be indexed;
> > if
> > > > you
> > > > > > > > create a
> > > > > > > > > > >> table
> > > > > > > > > > >> > with a foreign key constraint but no index on a
> given
> > > > > column,
> > > > > > an
> > > > > > > > > index
> > > > > > > > > > >> is
> > > > > > > > > > >> > created. *
> > > > > > > > > > >> >
> > > > > > > > > > >> > So I asked Martin to especially create an index for
> > > > > > > > > > process_instance_id
> > > > > > > > > > >> > column on nodes table
> > > > > > > > > > >> > I think that will fix the problem detected on the
> > thread
> > > > > dump.
> > > > > > > > > > >> > The simpler process test to verify queries are fine
> > > still
> > > > > > > stands,
> > > > > > > > > > >> though ;)
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <
> > > > > > > tzimanyi@apache.org
> > > > > > > > >
> > > > > > > > > > >> wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> > > I always preferred pure JDBC over Hibernate
> myself,
> > > just
> > > > > for
> > > > > > > the
> > > > > > > > > > sake
> > > > > > > > > > >> of
> > > > > > > > > > >> > > control of what is happening :) So I would not -1
> > that
> > > > > > myself.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Tibor
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado
> > > Sarti <
> > > > > > > > > > >> > > ftirados@redhat.com>
> > > > > > > > > > >> > > napísal(a):
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > Hi,
> > > > > > > > > > >> > > > Usually I do not want to talk about work in
> > progress
> > > > > > because
> > > > > > > > > > >> > preliminary
> > > > > > > > > > >> > > > conclusions are pretty volatile but, well, there
> > > are a
> > > > > > > couple
> > > > > > > > of
> > > > > > > > > > >> things
> > > > > > > > > > >> > > > that can be concluded from the really valuable
> > > > > information
> > > > > > > > that
> > > > > > > > > > >> Martin
> > > > > > > > > > >> > > > provided.
> > > > > > > > > > >> > > > 1) In order to be able to determine if the
> number
> > of
> > > > > > > > statements
> > > > > > > > > is
> > > > > > > > > > >> > larger
> > > > > > > > > > >> > > > than expected, I asked Martin to test with a
> > simpler
> > > > > > process
> > > > > > > > > > >> > definition.
> > > > > > > > > > >> > > > One with just three nodes: start, script and
> end.
> > > The
> > > > > > script
> > > > > > > > one
> > > > > > > > > > >> should
> > > > > > > > > > >> > > > change just one variable. This way we can
> analyze
> > if
> > > > the
> > > > > > > > number
> > > > > > > > > of
> > > > > > > > > > >> > > queries
> > > > > > > > > > >> > > > is the expected one. From the single log (audit
> > was
> > > > > > > activated
> > > > > > > > > > them)
> > > > > > > > > > >> my
> > > > > > > > > > >> > > > conclusion is that the number of insert/updates
> > over
> > > > > > > processes
> > > > > > > > > and
> > > > > > > > > > >> > nodes
> > > > > > > > > > >> > > > (there a lot over task, that I will prefer to
> skip
> > > for
> > > > > > now,
> > > > > > > > baby
> > > > > > > > > > >> steps)
> > > > > > > > > > >> > > is
> > > > > > > > > > >> > > > the expected one.
> > > > > > > > > > >> > > > 2) Analysing the thread dump, we see around 15
> > > threads
> > > > > > > > executing
> > > > > > > > > > >> this
> > > > > > > > > > >> > > line
> > > > > > > > > > >> > > > at
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > > > > > > > > >> > > > so its pretty clear the code to be optimized ;).
> > I'm
> > > > > > > > evaluating
> > > > > > > > > > >> > > > possibilities within JPA/Hibernate, but I'm
> > starting
> > > > to
> > > > > > > think
> > > > > > > > > that
> > > > > > > > > > >> it
> > > > > > > > > > >> > > might
> > > > > > > > > > >> > > > be better to switch to JDBC and skip hibernate.
> > Our
> > > > > lives
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > > >> > > simpler,
> > > > > > > > > > >> > > > especially with a schema relatively simple like
> > ours
> > > > > (that
> > > > > > > > will
> > > > > > > > > be
> > > > > > > > > > >> my
> > > > > > > > > > >> > > > recommendation if I was an external consultant)
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <
> > > > > > > > > tzimanyi@apache.org
> > > > > > > > > > >
> > > > > > > > > > >> > > wrote:
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > > Hi,
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > this will be a bit off-topic. However as far
> as
> > > > > > > > performance, I
> > > > > > > > > > >> think
> > > > > > > > > > >> > we
> > > > > > > > > > >> > > > > should think about that we have string primary
> > > keys
> > > > > > > (IDs). I
> > > > > > > > > > would
> > > > > > > > > > >> > > expect
> > > > > > > > > > >> > > > > the database systems are much better with
> > indexing
> > > > > > numeric
> > > > > > > > > keys
> > > > > > > > > > >> than
> > > > > > > > > > >> > > > > strings. I remember from the past, when I was
> > > > working
> > > > > > with
> > > > > > > > > DBs,
> > > > > > > > > > >> that
> > > > > > > > > > >> > > > using
> > > > > > > > > > >> > > > > strings as keys or indexes was a discouraged
> > > > practice.
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > Best regards,
> > > > > > > > > > >> > > > > Tibor
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler
> > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > >
> > > > > > > > > > >> > > > > napísal(a):
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > > I changed the test to use MongoDB [1] and I
> > > don't
> > > > > see
> > > > > > a
> > > > > > > > > > >> performance
> > > > > > > > > > >> > > > > > degradation with this setup [2].
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > Please keep us posted of your findings.
> > Thanks!
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > Martin
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > [1]
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> >
> > > > > > > > > >
> > > > > > >
> > > >
> https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > > > > > > > >> > > > > > [2]
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > ________________________________________
> > > > > > > > > > >> > > > > > From: Francisco Javier Tirado Sarti <
> > > > > > > ftirados@redhat.com>
> > > > > > > > > > >> > > > > > Sent: Wednesday, February 7, 2024 11:40 AM
> > > > > > > > > > >> > > > > > To: dev@kie.apache.org
> > > > > > > > > > >> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION]
> > Performance
> > > > > > issues
> > > > > > > > with
> > > > > > > > > > >> > > data-index
> > > > > > > > > > >> > > > > > persistence addon
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > yes, it can be index degradation because of
> > > size,
> > > > > but
> > > > > > I
> > > > > > > > > > believe
> > > > > > > > > > >> (I
> > > > > > > > > > >> > > > might
> > > > > > > > > > >> > > > > be
> > > > > > > > > > >> > > > > > wrong) the db is too small (yet) for that.
> > > > > > > > > > >> > > > > > But, eventually, Postgres, when the DB is
> huge
> > > > > enough,
> > > > > > > > > > >> unavoidably
> > > > > > > > > > >> > > will
> > > > > > > > > > >> > > > > > behave like the graphic that Martin sent.
> > > > > > > > > > >> > > > > > Since I believe we are not huge enough
> (yet),
> > > lets
> > > > > > rule
> > > > > > > > out
> > > > > > > > > > >> another
> > > > > > > > > > >> > > > issue
> > > > > > > > > > >> > > > > > by analysing the sql logs (I requested those
> > to
> > > > > Martin
> > > > > > > > > offline
> > > > > > > > > > >> and
> > > > > > > > > > >> > he
> > > > > > > > > > >> > > > is
> > > > > > > > > > >> > > > > > going to kindly collect them).
> > > > > > > > > > >> > > > > > Also Im curious to know if Mongo behave in
> the
> > > > same
> > > > > > way.
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique
> > Gonzalez
> > > > > > > Martinez <
> > > > > > > > > > >> > > > > > egonzalez@apache.org> wrote:
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > > Hi Francisco,
> > > > > > > > > > >> > > > > > > I would highly recommend to check indexes
> > and
> > > > how
> > > > > > the
> > > > > > > > > > updates
> > > > > > > > > > >> > work
> > > > > > > > > > >> > > in
> > > > > > > > > > >> > > > > > data
> > > > > > > > > > >> > > > > > > index to avoid full scan table and lock
> the
> > > full
> > > > > > > table.
> > > > > > > > > Some
> > > > > > > > > > >> db
> > > > > > > > > > >> > are
> > > > > > > > > > >> > > > > very
> > > > > > > > > > >> > > > > > > sensitive to that.
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > El mié, 7 feb 2024, 18:41, Francisco
> Javier
> > > > Tirado
> > > > > > > > Sarti <
> > > > > > > > > > >> > > > > > > ftirados@redhat.com> escribió:
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > > Hi Martin,
> > > > > > > > > > >> > > > > > > > While I analyze the data, let me ask you
> > if
> > > it
> > > > > is
> > > > > > > > > possible
> > > > > > > > > > >> to
> > > > > > > > > > >> > > > perform
> > > > > > > > > > >> > > > > > > > another check (similar in a way to
> > disabling
> > > > > > > > data-index
> > > > > > > > > > like
> > > > > > > > > > >> > you
> > > > > > > > > > >> > > > do)
> > > > > > > > > > >> > > > > > Can
> > > > > > > > > > >> > > > > > > > you switch to MongoDB persistence and
> > check
> > > if
> > > > > the
> > > > > > > > same
> > > > > > > > > > >> > > degradation
> > > > > > > > > > >> > > > > > that
> > > > > > > > > > >> > > > > > > is
> > > > > > > > > > >> > > > > > > > there for postgres remains?
> > > > > > > > > > >> > > > > > > > I do not know if this is feasible but
> will
> > > > > > certainly
> > > > > > > > > > >> indicate
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > > > problem
> > > > > > > > > > >> > > > > > > > is on the postgres storage layer and I
> do
> > > not
> > > > > > have a
> > > > > > > > > clear
> > > > > > > > > > >> > > > prediction
> > > > > > > > > > >> > > > > > of
> > > > > > > > > > >> > > > > > > > what we will see when doing this switch.
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin
> > Weiler
> > > > > > > > > > >> > > > <mweiler@ibm.com.invalid
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > > > wrote:
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > > Hi Francisco,
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > thanks for your work on this important
> > > > topic!
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > I would like to share some test
> results
> > > > here,
> > > > > > > which
> > > > > > > > > > might
> > > > > > > > > > >> > help
> > > > > > > > > > >> > > to
> > > > > > > > > > >> > > > > > > improve
> > > > > > > > > > >> > > > > > > > > the codebase even further. I am using
> > the
> > > > > jmeter
> > > > > > > > based
> > > > > > > > > > >> test
> > > > > > > > > > >> > > case
> > > > > > > > > > >> > > > > from
> > > > > > > > > > >> > > > > > > > Pere
> > > > > > > > > > >> > > > > > > > > and Enrique (thanks guys!) [1] which
> > uses
> > > a
> > > > > load
> > > > > > > of
> > > > > > > > 30
> > > > > > > > > > >> > threads
> > > > > > > > > > >> > > to
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > 1) start a new process instance (POST)
> > > > > > > > > > >> > > > > > > > > 2) retrieve tasks for a user (GET)
> > > > > > > > > > >> > > > > > > > > 3) fetches task details (GET)
> > > > > > > > > > >> > > > > > > > > 4) complete a task (POST)
> > > > > > > > > > >> > > > > > > > > 5) execute a query on data-audit
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > With this test setup, I noticed that
> the
> > > > > > > performance
> > > > > > > > > for
> > > > > > > > > > >> the
> > > > > > > > > > >> > > POST
> > > > > > > > > > >> > > > > > > > > requests, in particular the one to
> > start a
> > > > new
> > > > > > > > process
> > > > > > > > > > >> > > instance,
> > > > > > > > > > >> > > > > > > degrades
> > > > > > > > > > >> > > > > > > > > over time - see graph [2]. If I run
> the
> > > same
> > > > > > test
> > > > > > > > > > without
> > > > > > > > > > >> > > > > data-index,
> > > > > > > > > > >> > > > > > > > then
> > > > > > > > > > >> > > > > > > > > there is no such performance
> degradation
> > > > [3].
> > > > > > You
> > > > > > > > can
> > > > > > > > > > >> find a
> > > > > > > > > > >> > > > thread
> > > > > > > > > > >> > > > > > > dump
> > > > > > > > > > >> > > > > > > > > captured a few minutes into the first
> > test
> > > > > here
> > > > > > > [4]
> > > > > > > > > that
> > > > > > > > > > >> > might
> > > > > > > > > > >> > > > help
> > > > > > > > > > >> > > > > > to
> > > > > > > > > > >> > > > > > > > see
> > > > > > > > > > >> > > > > > > > > some of the contention points.
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > I'd appreciate if you could take a
> look
> > > and
> > > > > see
> > > > > > if
> > > > > > > > > there
> > > > > > > > > > >> is
> > > > > > > > > > >> > > > > something
> > > > > > > > > > >> > > > > > > > that
> > > > > > > > > > >> > > > > > > > > can be further improved based on your
> > > > previous
> > > > > > > work.
> > > > > > > > > If
> > > > > > > > > > >> you
> > > > > > > > > > >> > > need
> > > > > > > > > > >> > > > > any
> > > > > > > > > > >> > > > > > > > > additional data, let me know, but
> > > otherwise
> > > > it
> > > > > > is
> > > > > > > > > > >> > > straightforward
> > > > > > > > > > >> > > > > to
> > > > > > > > > > >> > > > > > > run
> > > > > > > > > > >> > > > > > > > > the jmeter test as well.
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > Thanks,
> > > > > > > > > > >> > > > > > > > > Martin
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > [1]
> > > > > > > > > > >>
> https://github.com/pefernan/job-service-refactor-test/
> > > > > > > > > > >> > > > > > > > > [2]
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > > > > > > >> > > > > > > > > [3]
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > > > > > > >> > > > > > > > > [4]
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > >
> ________________________________________
> > > > > > > > > > >> > > > > > > > > From: Francisco Javier Tirado Sarti <
> > > > > > > > > > ftirados@redhat.com>
> > > > > > > > > > >> > > > > > > > > Sent: Wednesday, January 17, 2024 9:13
> > AM
> > > > > > > > > > >> > > > > > > > > To: dev@kie.apache.org
> > > > > > > > > > >> > > > > > > > > Cc: Pere Fernandez Perez
> > > > > > > > > > >> > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION]
> > > > > Performance
> > > > > > > > > issues
> > > > > > > > > > >> with
> > > > > > > > > > >> > > > > > data-index
> > > > > > > > > > >> > > > > > > > > persistence addon
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > Hi Alex,
> > > > > > > > > > >> > > > > > > > > I did not take times (which depends
> on a
> > > > > number
> > > > > > of
> > > > > > > > > > >> variables
> > > > > > > > > > >> > > that
> > > > > > > > > > >> > > > > > > > > drastically change between
> > environments),
> > > > but
> > > > > > > verify
> > > > > > > > > > that
> > > > > > > > > > >> the
> > > > > > > > > > >> > > > > number
> > > > > > > > > > >> > > > > > of
> > > > > > > > > > >> > > > > > > > > updates has been reduced drastically
> > > without
> > > > > > > losing
> > > > > > > > > > >> > > > functionality,
> > > > > > > > > > >> > > > > > > which
> > > > > > > > > > >> > > > > > > > is
> > > > > > > > > > >> > > > > > > > > objectively a good thing. If before
> the
> > > > > change,
> > > > > > > for
> > > > > > > > > > every
> > > > > > > > > > >> > node
> > > > > > > > > > >> > > > > > > executed,
> > > > > > > > > > >> > > > > > > > we
> > > > > > > > > > >> > > > > > > > > have an update for every node
> previously
> > > > > > executed,
> > > > > > > > so
> > > > > > > > > > if a
> > > > > > > > > > >> > > > process
> > > > > > > > > > >> > > > > > have
> > > > > > > > > > >> > > > > > > > 50
> > > > > > > > > > >> > > > > > > > > nodes to execute, we were performing
> > > nearly
> > > > > > > 50*51/2
> > > > > > > > > > >> updates,
> > > > > > > > > > >> > > > which
> > > > > > > > > > >> > > > > > > gives
> > > > > > > > > > >> > > > > > > > us
> > > > > > > > > > >> > > > > > > > > a total of  1275 updates, now we have
> > just
> > > > one
> > > > > > for
> > > > > > > > > every
> > > > > > > > > > >> node
> > > > > > > > > > >> > > > being
> > > > > > > > > > >> > > > > > > > > executed, implying a total of 50
> > updates.
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex
> > > > Porcelli
> > > > > <
> > > > > > > > > > >> > > alex@porcelli.me>
> > > > > > > > > > >> > > > > > > wrote:
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > Francisco,
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > I noticed that your PR has been
> > merged,
> > > > but
> > > > > I
> > > > > > > was
> > > > > > > > > > >> expecting
> > > > > > > > > > >> > > (at
> > > > > > > > > > >> > > > > > least
> > > > > > > > > > >> > > > > > > > > > was my understanding from this
> thread)
> > > > that
> > > > > > > before
> > > > > > > > > > >> merging
> > > > > > > > > > >> > > some
> > > > > > > > > > >> > > > > > > > > > benchmark data would be shared in
> > > advance
> > > > -
> > > > > to
> > > > > > > > > assess
> > > > > > > > > > >> the
> > > > > > > > > > >> > > > > > > cost/benefit
> > > > > > > > > > >> > > > > > > > > > of such a decent size change.
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > Do you have any information to
> share?
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM
> > > Francisco
> > > > > > Javier
> > > > > > > > > > Tirado
> > > > > > > > > > >> > Sarti
> > > > > > > > > > >> > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > Yes, as intended, now we have one
> > > select
> > > > > and
> > > > > > > one
> > > > > > > > > > >> > > > insert/update
> > > > > > > > > > >> > > > > > per
> > > > > > > > > > >> > > > > > > > node
> > > > > > > > > > >> > > > > > > > > > > event.
> > > > > > > > > > >> > > > > > > > > > > I moved the PR as ready for review
> > and
> > > > > give
> > > > > > > > @Pere
> > > > > > > > > > >> > Fernandez
> > > > > > > > > > >> > > > > Perez
> > > > > > > > > > >> > > > > > > > > > > <pe...@redhat.com> permission
> to
> > > the
> > > > > > > branch
> > > > > > > > so
> > > > > > > > > > he
> > > > > > > > > > >> can
> > > > > > > > > > >> > > > edit
> > > > > > > > > > >> > > > > it
> > > > > > > > > > >> > > > > > > in
> > > > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >> > > > > > > > > > > next two weeks (Ill be on PTO)  if
> > > > > desired,
> > > > > > > > before
> > > > > > > > > > >> > merging.
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM
> Alex
> > > > > > Porcelli
> > > > > > > <
> > > > > > > > > > >> > > > > alex@porcelli.me>
> > > > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > Cool, thank you Francisco!
> > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > Did you manage to get some
> > > preliminary
> > > > > > data
> > > > > > > > > about
> > > > > > > > > > >> > > > > improvements?
> > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM
> > > > > Francisco
> > > > > > > > > Javier
> > > > > > > > > > >> > Tirado
> > > > > > > > > > >> > > > > Sarti
> > > > > > > > > > >> > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > Yes, after some delay because
> of
> > > > > > quarkus 3
> > > > > > > > > > >> migration.
> > > > > > > > > > >> > > Im
> > > > > > > > > > >> > > > > > > refining
> > > > > > > > > > >> > > > > > > > > > this
> > > > > > > > > > >> > > > > > > > > > > > > draft PR
> > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > >
> > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > On Thu, Dec 21, 2023 at
> 5:48 PM
> > > Alex
> > > > > > > > Porcelli
> > > > > > > > > <
> > > > > > > > > > >> > > > > > > alex@porcelli.me>
> > > > > > > > > > >> > > > > > > > > > wrote:
> > > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > Any update or new findings
> on
> > > this
> > > > > > > topic?
> > > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > On Tue, Nov 28, 2023 at
> > 8:38 AM
> > > > > > > Francisco
> > > > > > > > > > Javier
> > > > > > > > > > >> > > Tirado
> > > > > > > > > > >> > > > > > Sarti
> > > > > > > > > > >> > > > > > > > > > > > > > <ft...@redhat.com>
> wrote:
> > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > > Hi Alex,
> > > > > > > > > > >> > > > > > > > > > > > > > > After considering
> different
> > > > > options
> > > > > > to
> > > > > > > > > > improve
> > > > > > > > > > >> > > > > > performance,
> > > > > > > > > > >> > > > > > > > we
> > > > > > > > > > >> > > > > > > > > > feel
> > > > > > > > > > >> > > > > > > > > > > > that
> > > > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > > > > >> > > > > > > > > > > > > > > is time to "partially"
> move
> > > away
> > > > > > from
> > > > > > > > the
> > > > > > > > > > >> current
> > > > > > > > > > >> > > Map
> > > > > > > > > > >> > > > > > style
> > > > > > > > > > >> > > > > > > > > > > > interface (
> > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > > > > > >> > > > > > > > > > > > > > )
> > > > > > > > > > >> > > > > > > > > > > > > > > which was shared with
> > Trusty,
> > > to
> > > > > one
> > > > > > > > more
> > > > > > > > > > >> > suitable
> > > > > > > > > > >> > > > for
> > > > > > > > > > >> > > > > > > usage
> > > > > > > > > > >> > > > > > > > > > with a
> > > > > > > > > > >> > > > > > > > > > > > > > > relational DB like
> > postgresql
> > > > (but
> > > > > > > still
> > > > > > > > > > >> > compatible
> > > > > > > > > > >> > > > > with
> > > > > > > > > > >> > > > > > > big
> > > > > > > > > > >> > > > > > > > > > table
> > > > > > > > > > >> > > > > > > > > > > > dbs).
> > > > > > > > > > >> > > > > > > > > > > > > > > The idea will be to
> replace
> > > > > generic
> > > > > > > > > Storage
> > > > > > > > > > >> > > interface
> > > > > > > > > > >> > > > > by
> > > > > > > > > > >> > > > > > > four
> > > > > > > > > > >> > > > > > > > > > > > specific
> > > > > > > > > > >> > > > > > > > > > > > > > > interfaces (which will
> > inherit
> > > > > from
> > > > > > a
> > > > > > > > > common
> > > > > > > > > > >> one
> > > > > > > > > > >> > > that
> > > > > > > > > > >> > > > > > keeps
> > > > > > > > > > >> > > > > > > > the
> > > > > > > > > > >> > > > > > > > > > query
> > > > > > > > > > >> > > > > > > > > > > > > > part
> > > > > > > > > > >> > > > > > > > > > > > > > > at is it. with get and
> query
> > > > > > methods),
> > > > > > > > > that
> > > > > > > > > > >> will
> > > > > > > > > > >> > > > > include
> > > > > > > > > > >> > > > > > > the
> > > > > > > > > > >> > > > > > > > > > required
> > > > > > > > > > >> > > > > > > > > > > > > > > modification operations
> for
> > > the
> > > > > four
> > > > > > > > > > DataIndex
> > > > > > > > > > >> > > > > "domains":
> > > > > > > > > > >> > > > > > > > > > > > > > processinstance,
> > > > > > > > > > >> > > > > > > > > > > > > > > usertask,
> processdefinitions
> > > and
> > > > > > jobs.
> > > > > > > > > Those
> > > > > > > > > > >> > > > interfaces
> > > > > > > > > > >> > > > > > > will
> > > > > > > > > > >> > > > > > > > > > define
> > > > > > > > > > >> > > > > > > > > > > > > > methods
> > > > > > > > > > >> > > > > > > > > > > > > > > like addNode, addVariable,
> > > > > > updateTask,
> > > > > > > > > > >> > > > > addAttachment.....
> > > > > > > > > > >> > > > > > > > that
> > > > > > > > > > >> > > > > > > > > > will
> > > > > > > > > > >> > > > > > > > > > > > allow
> > > > > > > > > > >> > > > > > > > > > > > > > > the persistent layer
> > > > > implementation
> > > > > > > to
> > > > > > > > > just
> > > > > > > > > > >> > update
> > > > > > > > > > >> > > > the
> > > > > > > > > > >> > > > > > > > needed
> > > > > > > > > > >> > > > > > > > > > info
> > > > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > > >> > > > > > > > > > > > > > > DB  (for example, for
> > addNode
> > > in
> > > > > > > > Postgres,
> > > > > > > > > > >> just
> > > > > > > > > > >> > > > insert
> > > > > > > > > > >> > > > > a
> > > > > > > > > > >> > > > > > > row
> > > > > > > > > > >> > > > > > > > > into
> > > > > > > > > > >> > > > > > > > > > > > nodes
> > > > > > > > > > >> > > > > > > > > > > > > > > table, for addNode in
> Mongo,
> > > > > > basically
> > > > > > > > the
> > > > > > > > > > >> same
> > > > > > > > > > >> > > > atomic
> > > > > > > > > > >> > > > > > > upsert
> > > > > > > > > > >> > > > > > > > > > > > operation
> > > > > > > > > > >> > > > > > > > > > > > > > > that is currently done).
> > > > > Therefore,
> > > > > > we
> > > > > > > > > > >> increase
> > > > > > > > > > >> > > > > > performance
> > > > > > > > > > >> > > > > > > > for
> > > > > > > > > > >> > > > > > > > > > > > Postgres
> > > > > > > > > > >> > > > > > > > > > > > > > > and keep the current one
> for
> > > > > Mongo.
> > > > > > > The
> > > > > > > > > > >> current
> > > > > > > > > > >> > DB
> > > > > > > > > > >> > > > > > schemas
> > > > > > > > > > >> > > > > > > > > won't
> > > > > > > > > > >> > > > > > > > > > be
> > > > > > > > > > >> > > > > > > > > > > > > > > touched.
> > > > > > > > > > >> > > > > > > > > > > > > > > Since the code change is
> > > large,
> > > > I
> > > > > do
> > > > > > > not
> > > > > > > > > > think
> > > > > > > > > > >> > I'll
> > > > > > > > > > >> > > > be
> > > > > > > > > > >> > > > > > able
> > > > > > > > > > >> > > > > > > > to
> > > > > > > > > > >> > > > > > > > > > have
> > > > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > > >> > > > > > > > > > > > > > PR
> > > > > > > > > > >> > > > > > > > > > > > > > > ready till next week.
> > > > > > > > > > >> > > > > > > > > > > > > > > But before starting,
> please
> > > let
> > > > me
> > > > > > > know
> > > > > > > > if
> > > > > > > > > > >> that
> > > > > > > > > > >> > > > > approach
> > > > > > > > > > >> > > > > > is
> > > > > > > > > > >> > > > > > > > > fine
> > > > > > > > > > >> > > > > > > > > > for
> > > > > > > > > > >> > > > > > > > > > > > you.
> > > > > > > > > > >> > > > > > > > > > > > > > > Best regards.
> > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> > > 6:55 PM
> > > > > Alex
> > > > > > > > > > Porcelli
> > > > > > > > > > >> <
> > > > > > > > > > >> > > > > > > > > alex@porcelli.me>
> > > > > > > > > > >> > > > > > > > > > > > wrote:
> > > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > > > Thank you Francisco to
> > > getting
> > > > > > > deeper
> > > > > > > > on
> > > > > > > > > > >> this…
> > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > > > Looking forward to see
> the
> > > > > results
> > > > > > > of
> > > > > > > > > your
> > > > > > > > > > >> > > > suggested
> > > > > > > > > > >> > > > > > > > > > improvements.
> > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> > > > 9:40 AM
> > > > > > > > > Francisco
> > > > > > > > > > >> > Javier
> > > > > > > > > > >> > > > > Tirado
> > > > > > > > > > >> > > > > > > > > Sarti <
> > > > > > > > > > >> > > > > > > > > > > > > > > > ftirados@redhat.com>
> > wrote:
> > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > > > > I forgot to attach the
> > > > queries
> > > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > > > > On Fri, Nov 24, 2023
> at
> > > > > 3:04 PM
> > > > > > > > > > Francisco
> > > > > > > > > > >> > > Javier
> > > > > > > > > > >> > > > > > Tirado
> > > > > > > > > > >> > > > > > > > > > Sarti <
> > > > > > > > > > >> > > > > > > > > > > > > > > > > ftirados@redhat.com>
> > > wrote:
> > > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > > > >> Hi,
> > > > > > > > > > >> > > > > > > > > > > > > > > > >> A brief update on
> this
> > > > topic.
> > > > > > > > > > >> > > > > > > > > > > > > > > > >> After doing a simple
> > test
> > > > > with
> > > > > > > > > example
> > > > > > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > > > > >> > > > > > > > > > > > > > > > ,
> > > > > > > > > > >> > > > > > > > > > > > > > > > >> the number of updates
> > > over
> > > > > > Nodes
> > > > > > > > > table
> > > > > > > > > > is
> > > > > > > > > > >> > n*n,
> > > > > > > > > > >> > > > so
> > > > > > > > > > >> > > > > we
> > > > > > > > > > >> > > > > > > > > manage
> > > > > > > > > > >> > > > > > > > > > to
> > > > > > > > > > >> > > > > > > > > > > > > > obtain a
> > > > > > > > > > >> > > > > > > > > > > > > > > > >> perfect quadratic
> > > > performance
> > > > > > > > > > >> degradation.
> > > > > > > > > > >> > The
> > > > > > > > > > >> > > > > > problem
> > > > > > > > > > >> > > > > > > > is
> > > > > > > > > > >> > > > > > > > > > worse
> > > > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > > >> > > > > > > > > > > > > > > > case
> > > > > > > > > > >> > > > > > > > > > > > > > > > >> of Serverless
> Workflow
> > > than
> > > > > in
> > > > > > > BPMN
> > > > > > > > > > >> because
> > > > > > > > > > >> > we
> > > > > > > > > > >> > > > the
> > > > > > > > > > >> > > > > > > > number
> > > > > > > > > > >> > > > > > > >

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Francisco Javier Tirado Sarti <ft...@redhat.com>]

Richard,
You can also disable event publishing (which is the mechanism to sincronize
runtimes with dataindex). In any case, since this mechanism is async, the
performance of the process execution won't be affected, even if enabled.

On Wed, Feb 21, 2024 at 11:24 AM Richard Bourner <ri...@gmail.com> wrote:

> Got it. Thanks Enrique.
>
>
> Le mer. 21 févr. 2024, 00:18, Enrique Gonzalez Martinez <
> egonzalez@apache.org> a écrit :
>
> > Regarding your STP concerns, they were addressed in my previous comment:
> >
> > STP is a concept, a process with certain constraints: no persistence and
> > returning the outcome in the call (sync execution with no idle states).
> It
> > was a requirement from a user in the past. One of the requirements was
> > leaving no trail. In v7 was easy because you could disable the audit in
> > that case. Actually we have the same way to do what we did in v7 in here
> as
> > you can add/remove index just removing dependencies.
> >
> > So in fact in microservice you just need to exclude the data index from
> the
> > app and you wont have data index.
> >
> >
> >
> > El mié, 21 feb 2024, 1:40, Richard Bourner <ri...@gmail.com> escribió:
> >
> > > +1 with Martin's email.
> > >
> > > One question though in regards to Martin's point #3 and to previous
> > > following statement from Francisco: "*... keeping finishing
> > > process instances "for a while" in DataIndex was the only way for users
> > to
> > > query the result of straight through processes*"
> > > --> Is this the only use case where data index would be needed for STP?
> > > I am asking because clients will already get their result in the JSON
> > > returned from the synchronous REST call, so adding an extra computing
> > time
> > > for data index persistence does not seem right to me in the context of
> > > decision services that are supposed to be very fast to return
> (typically
> > > rule tasks+scripts).
> > > Or is it that we also want to provide some GraphQL capabilities, even
> for
> > > STP use cases?
> > >
> > > Also, what would "*for a while*" mean exactly?  Will it be
> configurable?
> > > Will there be a default expiration value?
> > >
> > > I am assuming this is all work in progress, and you may not have
> answers
> > to
> > > all my questions, no problem with that.
> > >
> > > Thanks.
> > >
> > > On Tue, Feb 20, 2024 at 5:39 PM Martin Weiler <mweiler@ibm.com.invalid
> >
> > > wrote:
> > >
> > > > IMO, it is good to have this discussion around data sanity now
> instead
> > of
> > > > putting it off until later when data has already accumulated in
> > > production
> > > > environments.
> > > >
> > > > Based on the input here, we are dealing with three types of data:
> > > > 1. Runtime data - active instances only, engine cleans up the data
> > > > automatically at process instance end
> > > > 2. Historic log data - data created by data-audit intended for long
> > term
> > > > storage
> > > > 3. Data-index data - somehow this data falls in between the two
> > > > aforementioned categories, with the idea of the data being "recent",
> > but
> > > > not restricted to active instances only
> > > >
> > > > We'd need purge strategies for both #2 and #3 (perhaps different
> ones,
> > or
> > > > with different config settings) in order to prevent unlimited data
> > > growth.
> > > >
> > > > ________________________________________
> > > > From: Enrique Gonzalez Martinez <eg...@apache.org>
> > > > Sent: Monday, February 19, 2024 7:11 AM
> > > > To: dev@kie.apache.org
> > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> data-index
> > > > persistence addon
> > > >
> > > > Hi Francisco,
> > > > To give you more context about this.
> > > >
> > > > STP is a concept, a process with certain constraints: no persistence
> > and
> > > > returning the outcome in the call (sync execution with no idle
> states).
> > > It
> > > > was a requirement from a user in the past. One of the requirements
> was
> > > > leaving no trail. In v7 was easy because you could disable the audit
> in
> > > > that case. Actually we have the same way to do what we did in v7 in
> > here
> > > as
> > > > you can add/remove index just removing deps.
> > > >
> > > > We have the same outcome with different approaches and STP is already
> > > > delivered.
> > > >
> > > > El lun, 19 feb 2024 a las 14:46, Francisco Javier Tirado Sarti (<
> > > > ftirados@redhat.com>) escribió:
> > > >
> > > > > Regarding STP (which is not a concept that we have in the code. I
> > mean
> > > > STP
> > > > > are processes as nonSTP are), I guess, as all processes, they were
> > kept
> > > > in
> > > > > DataIndex once completed because users wanted (and still wants) to
> > > check
> > > > > the result once the call had been performed. If we want to leave no
> > > trace
> > > > > of them in DataIndex for some reason, we will need to make it a
> > > > > Runtimes concept so DataIndex can handle them in a different way.
> > > > >
> > > > > On Mon, Feb 19, 2024 at 2:27 PM Enrique Gonzalez Martinez <
> > > > > egonzalez@apache.org> wrote:
> > > > >
> > > > > > Alex:
> > > > > > Right now the data index is working in the same way as it did in
> v7
> > > > with
> > > > > > the emitters. The only difference between two impl is that in
> here
> > > the
> > > > > > storage is pgsql instead elastic search.  You are right regarding
> > is
> > > a
> > > > > > snapshot of the last state of the process but we did never define
> > how
> > > > > long
> > > > > > would be alive that dats Honestly i am happy right now with the
> way
> > > it
> > > > > > works. The clean up mechanism is still tbd because we still need
> to
> > > > > discuss
> > > > > > other stuff first.
> > > > > >
> > > > > >
> > > > > > Regarding stp is to leave no trail because u can get the outcome
> > > > directly
> > > > > > from the call. It was defined like that in v7. So there is no use
> > for
> > > > the
> > > > > > index or the audit.
> > > > > >
> > > > > > El lun, 19 feb 2024, 14:13, Francisco Javier Tirado Sarti <
> > > > > > ftirados@redhat.com> escribió:
> > > > > >
> > > > > > > Hi Alex,
> > > > > > > There has been some confusion about the purpose of DataIndex.
> To
> > be
> > > > > > honest
> > > > > > > I believe they were already sorted out, but your e-mail makes
> me
> > > > think
> > > > > > that
> > > > > > > is not the case ;). I let Kris to clarify that with you. My
> view
> > is
> > > > > that
> > > > > > > data-index is a way to query recently closed and active
> processes
> > > > (the
> > > > > > key
> > > > > > > here is the definition of recently, which in my opinion should
> be
> > > > > > > configurable)
> > > > > > > But, besides that discussion and being pragmatic, keeping
> > finishing
> > > > > > process
> > > > > > > instances "for a while" in DataIndex was the only way for users
> > to
> > > > > query
> > > > > > > the result of straight through processes. That's a function
> that
> > > > cannot
> > > > > > be
> > > > > > > removed right now
> > > > > > >
> > > > > > > On Mon, Feb 19, 2024 at 1:33 PM Alex Porcelli <
> > porcelli@apache.org
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > if data index was supposed to provide snapshot view of the
> > > process
> > > > > > > > instance… why do we keep it after the process instance is
> > > finished?
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Feb 19, 2024 at 7:12 AM Francisco Javier Tirado
> Sarti <
> > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > >
> > > > > > > > > Hi Martin.
> > > > > > > > > After taking a deeper look at this, I realize that the
> > > behaviour
> > > > is
> > > > > > the
> > > > > > > > > expected one.
> > > > > > > > > Runtimes DB does not track the completed process instance
> > > (that's
> > > > > > what
> > > > > > > > the
> > > > > > > > > JDBCProcessInstances warn is telling us), but DataIndex, as
> > > > > expected,
> > > > > > > is
> > > > > > > > > tracking it in processes and nodes table. And yes it will
> > grow
> > > > over
> > > > > > > time.
> > > > > > > > > What we need is some configurable purge mechanism for
> > > DataIndex,
> > > > so
> > > > > > it
> > > > > > > > > eventually removes older completed process instances.
> > > > > > > > >
> > > > > > > > > On Tue, Feb 13, 2024 at 12:59 PM Francisco Javier Tirado
> > Sarti
> > > <
> > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > >
> > > > > > > > > > Hi Martin,
> > > > > > > > > > Good catch!. Looks like the skipping performed for
> process
> > > > > > instances
> > > > > > > is
> > > > > > > > > > not applied to node instances. Something we definitely
> need
> > > to
> > > > > > review
> > > > > > > > on
> > > > > > > > > > the runtimes side.
> > > > > > > > > >
> > > > > > > > > > On Mon, Feb 12, 2024 at 11:59 PM Martin Weiler
> > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> On a somewhat related note, testing a simple workflow
> > (start
> > > > ->
> > > > > > > script
> > > > > > > > > >> node -> end), I see the following messages in the logs:
> > > > > > > > > >> 2024-02-12 22:49:50,493 28758dde544c WARN
> > > > > > > > > >>
> [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> > > > > > > > > >> (executor-thread-3) Skipping create of process instance
> > id:
> > > > > > > > > >> 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2
> > > > > > > > > >>
> > > > > > > > > >> So far, so good. And I'd expect to see no trace of this
> > > > process
> > > > > in
> > > > > > > the
> > > > > > > > > >> database if I don't have data audit enabled.
> > > > > > > > > >>
> > > > > > > > > >> However, the 'processes' table contains a row with
> > state=2,
> > > > with
> > > > > > > > related
> > > > > > > > > >> entries in the 'nodes' table. In a load test, I see
> these
> > > > tables
> > > > > > > grow
> > > > > > > > > >> significantly over time. Am I missing something to have
> > > these
> > > > > > > entries
> > > > > > > > > >> cleaned up automatically?
> > > > > > > > > >>
> > > > > > > > > >> ________________________________________
> > > > > > > > > >> From: Martin Weiler <mw...@ibm.com.INVALID>
> > > > > > > > > >> Sent: Monday, February 12, 2024 3:40 PM
> > > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > > >> Subject: [EXTERNAL] RE: [DISCUSSION] Performance issues
> > with
> > > > > > > > data-index
> > > > > > > > > >> persistence addon
> > > > > > > > > >>
> > > > > > > > > >> Thanks everyone for your input. Based on this
> discussion,
> > I
> > > > > opened
> > > > > > > the
> > > > > > > > > >> following PR:
> > > > > > > > > >>
> > > https://github.com/apache/incubator-kie-kogito-apps/pull/1985
> > > > > > > > > >>
> > > > > > > > > >> With this change, the performance seems to be stable
> over
> > > > time:
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
> > > > > > > > > >>
> > > > > > > > > >> Martin
> > > > > > > > > >>
> > > > > > > > > >> ________________________________________
> > > > > > > > > >> From: Gonzalo Muñoz <gm...@apache.org>
> > > > > > > > > >> Sent: Friday, February 9, 2024 9:42 AM
> > > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > > >> Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues
> > with
> > > > > > > > data-index
> > > > > > > > > >> persistence addon
> > > > > > > > > >>
> > > > > > > > > >> Great work Francisco,
> > > > > > > > > >> Martin, take a look at this link with some related tips
> > (in
> > > > case
> > > > > > you
> > > > > > > > > find
> > > > > > > > > >> it useful):
> > > > > > > > > >>
> > > > https://www.cybertec-postgresql.com/en/index-your-foreign-key/
> > > > > > > > > >>
> > > > > > > > > >> El vie, 9 feb 2024 a las 17:20, Francisco Javier Tirado
> > > Sarti
> > > > (<
> > > > > > > > > >> ftirados@redhat.com>) escribió:
> > > > > > > > > >>
> > > > > > > > > >> > For the moment being, we will keep JPA till we exhaust
> > all
> > > > > > > > > >> possibilities,
> > > > > > > > > >> > let's call switching from jpa to jdbc our hidden plan
> B
> > ;)
> > > > > > > > > >> > I already told Martin, but in order everyone to know,
> > just
> > > > > after
> > > > > > > > > writing
> > > > > > > > > >> > the previous email, I thought "what if Postgres is not
> > > > > > > automatically
> > > > > > > > > >> > indexing foreign keys like mysql?" and, eureka
> > > > > > > > > >> > Postgres doc
> > > > > > > > > >> >
> > > > https://www.postgresql.org/docs/current/ddl-constraints.html
> > > > > > > > > >> > Mysql doc
> > > > > > > > > >> >
> > > > > > >
> > > https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> > > > > > > > > >> > These are the relevant excerpt
> > > > > > > > > >> >
> > > > > > > > > >> > *Postgresql*
> > > > > > > > > >> > *A foreign key must reference columns that either are
> a
> > > > > primary
> > > > > > > key
> > > > > > > > or
> > > > > > > > > >> form
> > > > > > > > > >> > a unique constraint, or are columns from a non-partial
> > > > unique
> > > > > > > index.
> > > > > > > > > >> This
> > > > > > > > > >> > means that the referenced columns always have an index
> > to
> > > > > allow
> > > > > > > > > >> efficient
> > > > > > > > > >> > lookups on whether a referencing row has a match.
> Since
> > a
> > > > > DELETE
> > > > > > > of
> > > > > > > > a
> > > > > > > > > >> row
> > > > > > > > > >> > from the referenced table or an UPDATE of a referenced
> > > > column
> > > > > > will
> > > > > > > > > >> require
> > > > > > > > > >> > a scan of the referencing table for rows matching the
> > old
> > > > > value,
> > > > > > > it
> > > > > > > > is
> > > > > > > > > >> > often a good idea to index the referencing columns
> too.
> > > > > Because
> > > > > > > this
> > > > > > > > > is
> > > > > > > > > >> not
> > > > > > > > > >> > always needed, and there are many choices available on
> > how
> > > > to
> > > > > > > index,
> > > > > > > > > the
> > > > > > > > > >> > declaration of a foreign key constraint does not
> > > > automatically
> > > > > > > > create
> > > > > > > > > an
> > > > > > > > > >> > index on the referencing columns.*
> > > > > > > > > >> > *Mysql*
> > > > > > > > > >> > *MySQL requires that foreign key columns be indexed;
> if
> > > you
> > > > > > > create a
> > > > > > > > > >> table
> > > > > > > > > >> > with a foreign key constraint but no index on a given
> > > > column,
> > > > > an
> > > > > > > > index
> > > > > > > > > >> is
> > > > > > > > > >> > created. *
> > > > > > > > > >> >
> > > > > > > > > >> > So I asked Martin to especially create an index for
> > > > > > > > > process_instance_id
> > > > > > > > > >> > column on nodes table
> > > > > > > > > >> > I think that will fix the problem detected on the
> thread
> > > > dump.
> > > > > > > > > >> > The simpler process test to verify queries are fine
> > still
> > > > > > stands,
> > > > > > > > > >> though ;)
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <
> > > > > > tzimanyi@apache.org
> > > > > > > >
> > > > > > > > > >> wrote:
> > > > > > > > > >> >
> > > > > > > > > >> > > I always preferred pure JDBC over Hibernate myself,
> > just
> > > > for
> > > > > > the
> > > > > > > > > sake
> > > > > > > > > >> of
> > > > > > > > > >> > > control of what is happening :) So I would not -1
> that
> > > > > myself.
> > > > > > > > > >> > >
> > > > > > > > > >> > > Tibor
> > > > > > > > > >> > >
> > > > > > > > > >> > > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado
> > Sarti <
> > > > > > > > > >> > > ftirados@redhat.com>
> > > > > > > > > >> > > napísal(a):
> > > > > > > > > >> > >
> > > > > > > > > >> > > > Hi,
> > > > > > > > > >> > > > Usually I do not want to talk about work in
> progress
> > > > > because
> > > > > > > > > >> > preliminary
> > > > > > > > > >> > > > conclusions are pretty volatile but, well, there
> > are a
> > > > > > couple
> > > > > > > of
> > > > > > > > > >> things
> > > > > > > > > >> > > > that can be concluded from the really valuable
> > > > information
> > > > > > > that
> > > > > > > > > >> Martin
> > > > > > > > > >> > > > provided.
> > > > > > > > > >> > > > 1) In order to be able to determine if the number
> of
> > > > > > > statements
> > > > > > > > is
> > > > > > > > > >> > larger
> > > > > > > > > >> > > > than expected, I asked Martin to test with a
> simpler
> > > > > process
> > > > > > > > > >> > definition.
> > > > > > > > > >> > > > One with just three nodes: start, script and end.
> > The
> > > > > script
> > > > > > > one
> > > > > > > > > >> should
> > > > > > > > > >> > > > change just one variable. This way we can analyze
> if
> > > the
> > > > > > > number
> > > > > > > > of
> > > > > > > > > >> > > queries
> > > > > > > > > >> > > > is the expected one. From the single log (audit
> was
> > > > > > activated
> > > > > > > > > them)
> > > > > > > > > >> my
> > > > > > > > > >> > > > conclusion is that the number of insert/updates
> over
> > > > > > processes
> > > > > > > > and
> > > > > > > > > >> > nodes
> > > > > > > > > >> > > > (there a lot over task, that I will prefer to skip
> > for
> > > > > now,
> > > > > > > baby
> > > > > > > > > >> steps)
> > > > > > > > > >> > > is
> > > > > > > > > >> > > > the expected one.
> > > > > > > > > >> > > > 2) Analysing the thread dump, we see around 15
> > threads
> > > > > > > executing
> > > > > > > > > >> this
> > > > > > > > > >> > > line
> > > > > > > > > >> > > > at
> > > > > > > > > >> > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > > > > > > > >> > > > so its pretty clear the code to be optimized ;).
> I'm
> > > > > > > evaluating
> > > > > > > > > >> > > > possibilities within JPA/Hibernate, but I'm
> starting
> > > to
> > > > > > think
> > > > > > > > that
> > > > > > > > > >> it
> > > > > > > > > >> > > might
> > > > > > > > > >> > > > be better to switch to JDBC and skip hibernate.
> Our
> > > > lives
> > > > > > will
> > > > > > > > be
> > > > > > > > > >> > > simpler,
> > > > > > > > > >> > > > especially with a schema relatively simple like
> ours
> > > > (that
> > > > > > > will
> > > > > > > > be
> > > > > > > > > >> my
> > > > > > > > > >> > > > recommendation if I was an external consultant)
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <
> > > > > > > > tzimanyi@apache.org
> > > > > > > > > >
> > > > > > > > > >> > > wrote:
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > > Hi,
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > this will be a bit off-topic. However as far as
> > > > > > > performance, I
> > > > > > > > > >> think
> > > > > > > > > >> > we
> > > > > > > > > >> > > > > should think about that we have string primary
> > keys
> > > > > > (IDs). I
> > > > > > > > > would
> > > > > > > > > >> > > expect
> > > > > > > > > >> > > > > the database systems are much better with
> indexing
> > > > > numeric
> > > > > > > > keys
> > > > > > > > > >> than
> > > > > > > > > >> > > > > strings. I remember from the past, when I was
> > > working
> > > > > with
> > > > > > > > DBs,
> > > > > > > > > >> that
> > > > > > > > > >> > > > using
> > > > > > > > > >> > > > > strings as keys or indexes was a discouraged
> > > practice.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > Best regards,
> > > > > > > > > >> > > > > Tibor
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler
> > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > >
> > > > > > > > > >> > > > > napísal(a):
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > > I changed the test to use MongoDB [1] and I
> > don't
> > > > see
> > > > > a
> > > > > > > > > >> performance
> > > > > > > > > >> > > > > > degradation with this setup [2].
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > Please keep us posted of your findings.
> Thanks!
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > Martin
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > [1]
> > > > > > > > > >> > > > >
> > > > > > > > > >> >
> > > > > > > > >
> > > > > >
> > > https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > > > > > > >> > > > > > [2]
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > ________________________________________
> > > > > > > > > >> > > > > > From: Francisco Javier Tirado Sarti <
> > > > > > ftirados@redhat.com>
> > > > > > > > > >> > > > > > Sent: Wednesday, February 7, 2024 11:40 AM
> > > > > > > > > >> > > > > > To: dev@kie.apache.org
> > > > > > > > > >> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION]
> Performance
> > > > > issues
> > > > > > > with
> > > > > > > > > >> > > data-index
> > > > > > > > > >> > > > > > persistence addon
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > yes, it can be index degradation because of
> > size,
> > > > but
> > > > > I
> > > > > > > > > believe
> > > > > > > > > >> (I
> > > > > > > > > >> > > > might
> > > > > > > > > >> > > > > be
> > > > > > > > > >> > > > > > wrong) the db is too small (yet) for that.
> > > > > > > > > >> > > > > > But, eventually, Postgres, when the DB is huge
> > > > enough,
> > > > > > > > > >> unavoidably
> > > > > > > > > >> > > will
> > > > > > > > > >> > > > > > behave like the graphic that Martin sent.
> > > > > > > > > >> > > > > > Since I believe we are not huge enough (yet),
> > lets
> > > > > rule
> > > > > > > out
> > > > > > > > > >> another
> > > > > > > > > >> > > > issue
> > > > > > > > > >> > > > > > by analysing the sql logs (I requested those
> to
> > > > Martin
> > > > > > > > offline
> > > > > > > > > >> and
> > > > > > > > > >> > he
> > > > > > > > > >> > > > is
> > > > > > > > > >> > > > > > going to kindly collect them).
> > > > > > > > > >> > > > > > Also Im curious to know if Mongo behave in the
> > > same
> > > > > way.
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique
> Gonzalez
> > > > > > Martinez <
> > > > > > > > > >> > > > > > egonzalez@apache.org> wrote:
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > > Hi Francisco,
> > > > > > > > > >> > > > > > > I would highly recommend to check indexes
> and
> > > how
> > > > > the
> > > > > > > > > updates
> > > > > > > > > >> > work
> > > > > > > > > >> > > in
> > > > > > > > > >> > > > > > data
> > > > > > > > > >> > > > > > > index to avoid full scan table and lock the
> > full
> > > > > > table.
> > > > > > > > Some
> > > > > > > > > >> db
> > > > > > > > > >> > are
> > > > > > > > > >> > > > > very
> > > > > > > > > >> > > > > > > sensitive to that.
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > El mié, 7 feb 2024, 18:41, Francisco Javier
> > > Tirado
> > > > > > > Sarti <
> > > > > > > > > >> > > > > > > ftirados@redhat.com> escribió:
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > > Hi Martin,
> > > > > > > > > >> > > > > > > > While I analyze the data, let me ask you
> if
> > it
> > > > is
> > > > > > > > possible
> > > > > > > > > >> to
> > > > > > > > > >> > > > perform
> > > > > > > > > >> > > > > > > > another check (similar in a way to
> disabling
> > > > > > > data-index
> > > > > > > > > like
> > > > > > > > > >> > you
> > > > > > > > > >> > > > do)
> > > > > > > > > >> > > > > > Can
> > > > > > > > > >> > > > > > > > you switch to MongoDB persistence and
> check
> > if
> > > > the
> > > > > > > same
> > > > > > > > > >> > > degradation
> > > > > > > > > >> > > > > > that
> > > > > > > > > >> > > > > > > is
> > > > > > > > > >> > > > > > > > there for postgres remains?
> > > > > > > > > >> > > > > > > > I do not know if this is feasible but will
> > > > > certainly
> > > > > > > > > >> indicate
> > > > > > > > > >> > the
> > > > > > > > > >> > > > > > problem
> > > > > > > > > >> > > > > > > > is on the postgres storage layer and I do
> > not
> > > > > have a
> > > > > > > > clear
> > > > > > > > > >> > > > prediction
> > > > > > > > > >> > > > > > of
> > > > > > > > > >> > > > > > > > what we will see when doing this switch.
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin
> Weiler
> > > > > > > > > >> > > > <mweiler@ibm.com.invalid
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > > > wrote:
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > > Hi Francisco,
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > thanks for your work on this important
> > > topic!
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > I would like to share some test results
> > > here,
> > > > > > which
> > > > > > > > > might
> > > > > > > > > >> > help
> > > > > > > > > >> > > to
> > > > > > > > > >> > > > > > > improve
> > > > > > > > > >> > > > > > > > > the codebase even further. I am using
> the
> > > > jmeter
> > > > > > > based
> > > > > > > > > >> test
> > > > > > > > > >> > > case
> > > > > > > > > >> > > > > from
> > > > > > > > > >> > > > > > > > Pere
> > > > > > > > > >> > > > > > > > > and Enrique (thanks guys!) [1] which
> uses
> > a
> > > > load
> > > > > > of
> > > > > > > 30
> > > > > > > > > >> > threads
> > > > > > > > > >> > > to
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > 1) start a new process instance (POST)
> > > > > > > > > >> > > > > > > > > 2) retrieve tasks for a user (GET)
> > > > > > > > > >> > > > > > > > > 3) fetches task details (GET)
> > > > > > > > > >> > > > > > > > > 4) complete a task (POST)
> > > > > > > > > >> > > > > > > > > 5) execute a query on data-audit
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > With this test setup, I noticed that the
> > > > > > performance
> > > > > > > > for
> > > > > > > > > >> the
> > > > > > > > > >> > > POST
> > > > > > > > > >> > > > > > > > > requests, in particular the one to
> start a
> > > new
> > > > > > > process
> > > > > > > > > >> > > instance,
> > > > > > > > > >> > > > > > > degrades
> > > > > > > > > >> > > > > > > > > over time - see graph [2]. If I run the
> > same
> > > > > test
> > > > > > > > > without
> > > > > > > > > >> > > > > data-index,
> > > > > > > > > >> > > > > > > > then
> > > > > > > > > >> > > > > > > > > there is no such performance degradation
> > > [3].
> > > > > You
> > > > > > > can
> > > > > > > > > >> find a
> > > > > > > > > >> > > > thread
> > > > > > > > > >> > > > > > > dump
> > > > > > > > > >> > > > > > > > > captured a few minutes into the first
> test
> > > > here
> > > > > > [4]
> > > > > > > > that
> > > > > > > > > >> > might
> > > > > > > > > >> > > > help
> > > > > > > > > >> > > > > > to
> > > > > > > > > >> > > > > > > > see
> > > > > > > > > >> > > > > > > > > some of the contention points.
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > I'd appreciate if you could take a look
> > and
> > > > see
> > > > > if
> > > > > > > > there
> > > > > > > > > >> is
> > > > > > > > > >> > > > > something
> > > > > > > > > >> > > > > > > > that
> > > > > > > > > >> > > > > > > > > can be further improved based on your
> > > previous
> > > > > > work.
> > > > > > > > If
> > > > > > > > > >> you
> > > > > > > > > >> > > need
> > > > > > > > > >> > > > > any
> > > > > > > > > >> > > > > > > > > additional data, let me know, but
> > otherwise
> > > it
> > > > > is
> > > > > > > > > >> > > straightforward
> > > > > > > > > >> > > > > to
> > > > > > > > > >> > > > > > > run
> > > > > > > > > >> > > > > > > > > the jmeter test as well.
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > Thanks,
> > > > > > > > > >> > > > > > > > > Martin
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > [1]
> > > > > > > > > >> https://github.com/pefernan/job-service-refactor-test/
> > > > > > > > > >> > > > > > > > > [2]
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > > > > > >> > > > > > > > > [3]
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > > > > > >> > > > > > > > > [4]
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > ________________________________________
> > > > > > > > > >> > > > > > > > > From: Francisco Javier Tirado Sarti <
> > > > > > > > > ftirados@redhat.com>
> > > > > > > > > >> > > > > > > > > Sent: Wednesday, January 17, 2024 9:13
> AM
> > > > > > > > > >> > > > > > > > > To: dev@kie.apache.org
> > > > > > > > > >> > > > > > > > > Cc: Pere Fernandez Perez
> > > > > > > > > >> > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION]
> > > > Performance
> > > > > > > > issues
> > > > > > > > > >> with
> > > > > > > > > >> > > > > > data-index
> > > > > > > > > >> > > > > > > > > persistence addon
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > Hi Alex,
> > > > > > > > > >> > > > > > > > > I did not take times (which depends on a
> > > > number
> > > > > of
> > > > > > > > > >> variables
> > > > > > > > > >> > > that
> > > > > > > > > >> > > > > > > > > drastically change between
> environments),
> > > but
> > > > > > verify
> > > > > > > > > that
> > > > > > > > > >> the
> > > > > > > > > >> > > > > number
> > > > > > > > > >> > > > > > of
> > > > > > > > > >> > > > > > > > > updates has been reduced drastically
> > without
> > > > > > losing
> > > > > > > > > >> > > > functionality,
> > > > > > > > > >> > > > > > > which
> > > > > > > > > >> > > > > > > > is
> > > > > > > > > >> > > > > > > > > objectively a good thing. If before the
> > > > change,
> > > > > > for
> > > > > > > > > every
> > > > > > > > > >> > node
> > > > > > > > > >> > > > > > > executed,
> > > > > > > > > >> > > > > > > > we
> > > > > > > > > >> > > > > > > > > have an update for every node previously
> > > > > executed,
> > > > > > > so
> > > > > > > > > if a
> > > > > > > > > >> > > > process
> > > > > > > > > >> > > > > > have
> > > > > > > > > >> > > > > > > > 50
> > > > > > > > > >> > > > > > > > > nodes to execute, we were performing
> > nearly
> > > > > > 50*51/2
> > > > > > > > > >> updates,
> > > > > > > > > >> > > > which
> > > > > > > > > >> > > > > > > gives
> > > > > > > > > >> > > > > > > > us
> > > > > > > > > >> > > > > > > > > a total of  1275 updates, now we have
> just
> > > one
> > > > > for
> > > > > > > > every
> > > > > > > > > >> node
> > > > > > > > > >> > > > being
> > > > > > > > > >> > > > > > > > > executed, implying a total of 50
> updates.
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex
> > > Porcelli
> > > > <
> > > > > > > > > >> > > alex@porcelli.me>
> > > > > > > > > >> > > > > > > wrote:
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > > Francisco,
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > I noticed that your PR has been
> merged,
> > > but
> > > > I
> > > > > > was
> > > > > > > > > >> expecting
> > > > > > > > > >> > > (at
> > > > > > > > > >> > > > > > least
> > > > > > > > > >> > > > > > > > > > was my understanding from this thread)
> > > that
> > > > > > before
> > > > > > > > > >> merging
> > > > > > > > > >> > > some
> > > > > > > > > >> > > > > > > > > > benchmark data would be shared in
> > advance
> > > -
> > > > to
> > > > > > > > assess
> > > > > > > > > >> the
> > > > > > > > > >> > > > > > > cost/benefit
> > > > > > > > > >> > > > > > > > > > of such a decent size change.
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > Do you have any information to share?
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM
> > Francisco
> > > > > Javier
> > > > > > > > > Tirado
> > > > > > > > > >> > Sarti
> > > > > > > > > >> > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > Yes, as intended, now we have one
> > select
> > > > and
> > > > > > one
> > > > > > > > > >> > > > insert/update
> > > > > > > > > >> > > > > > per
> > > > > > > > > >> > > > > > > > node
> > > > > > > > > >> > > > > > > > > > > event.
> > > > > > > > > >> > > > > > > > > > > I moved the PR as ready for review
> and
> > > > give
> > > > > > > @Pere
> > > > > > > > > >> > Fernandez
> > > > > > > > > >> > > > > Perez
> > > > > > > > > >> > > > > > > > > > > <pe...@redhat.com> permission to
> > the
> > > > > > branch
> > > > > > > so
> > > > > > > > > he
> > > > > > > > > >> can
> > > > > > > > > >> > > > edit
> > > > > > > > > >> > > > > it
> > > > > > > > > >> > > > > > > in
> > > > > > > > > >> > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > next two weeks (Ill be on PTO)  if
> > > > desired,
> > > > > > > before
> > > > > > > > > >> > merging.
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex
> > > > > Porcelli
> > > > > > <
> > > > > > > > > >> > > > > alex@porcelli.me>
> > > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > Cool, thank you Francisco!
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > Did you manage to get some
> > preliminary
> > > > > data
> > > > > > > > about
> > > > > > > > > >> > > > > improvements?
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM
> > > > Francisco
> > > > > > > > Javier
> > > > > > > > > >> > Tirado
> > > > > > > > > >> > > > > Sarti
> > > > > > > > > >> > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > Yes, after some delay because of
> > > > > quarkus 3
> > > > > > > > > >> migration.
> > > > > > > > > >> > > Im
> > > > > > > > > >> > > > > > > refining
> > > > > > > > > >> > > > > > > > > > this
> > > > > > > > > >> > > > > > > > > > > > > draft PR
> > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > >
> https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > On Thu, Dec 21, 2023 at 5:48 PM
> > Alex
> > > > > > > Porcelli
> > > > > > > > <
> > > > > > > > > >> > > > > > > alex@porcelli.me>
> > > > > > > > > >> > > > > > > > > > wrote:
> > > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > Any update or new findings on
> > this
> > > > > > topic?
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > On Tue, Nov 28, 2023 at
> 8:38 AM
> > > > > > Francisco
> > > > > > > > > Javier
> > > > > > > > > >> > > Tirado
> > > > > > > > > >> > > > > > Sarti
> > > > > > > > > >> > > > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > Hi Alex,
> > > > > > > > > >> > > > > > > > > > > > > > > After considering different
> > > > options
> > > > > to
> > > > > > > > > improve
> > > > > > > > > >> > > > > > performance,
> > > > > > > > > >> > > > > > > > we
> > > > > > > > > >> > > > > > > > > > feel
> > > > > > > > > >> > > > > > > > > > > > that
> > > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > > > >> > > > > > > > > > > > > > > is time to "partially" move
> > away
> > > > > from
> > > > > > > the
> > > > > > > > > >> current
> > > > > > > > > >> > > Map
> > > > > > > > > >> > > > > > style
> > > > > > > > > >> > > > > > > > > > > > interface (
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > > > > >> > > > > > > > > > > > > > )
> > > > > > > > > >> > > > > > > > > > > > > > > which was shared with
> Trusty,
> > to
> > > > one
> > > > > > > more
> > > > > > > > > >> > suitable
> > > > > > > > > >> > > > for
> > > > > > > > > >> > > > > > > usage
> > > > > > > > > >> > > > > > > > > > with a
> > > > > > > > > >> > > > > > > > > > > > > > > relational DB like
> postgresql
> > > (but
> > > > > > still
> > > > > > > > > >> > compatible
> > > > > > > > > >> > > > > with
> > > > > > > > > >> > > > > > > big
> > > > > > > > > >> > > > > > > > > > table
> > > > > > > > > >> > > > > > > > > > > > dbs).
> > > > > > > > > >> > > > > > > > > > > > > > > The idea will be to replace
> > > > generic
> > > > > > > > Storage
> > > > > > > > > >> > > interface
> > > > > > > > > >> > > > > by
> > > > > > > > > >> > > > > > > four
> > > > > > > > > >> > > > > > > > > > > > specific
> > > > > > > > > >> > > > > > > > > > > > > > > interfaces (which will
> inherit
> > > > from
> > > > > a
> > > > > > > > common
> > > > > > > > > >> one
> > > > > > > > > >> > > that
> > > > > > > > > >> > > > > > keeps
> > > > > > > > > >> > > > > > > > the
> > > > > > > > > >> > > > > > > > > > query
> > > > > > > > > >> > > > > > > > > > > > > > part
> > > > > > > > > >> > > > > > > > > > > > > > > at is it. with get and query
> > > > > methods),
> > > > > > > > that
> > > > > > > > > >> will
> > > > > > > > > >> > > > > include
> > > > > > > > > >> > > > > > > the
> > > > > > > > > >> > > > > > > > > > required
> > > > > > > > > >> > > > > > > > > > > > > > > modification operations for
> > the
> > > > four
> > > > > > > > > DataIndex
> > > > > > > > > >> > > > > "domains":
> > > > > > > > > >> > > > > > > > > > > > > > processinstance,
> > > > > > > > > >> > > > > > > > > > > > > > > usertask, processdefinitions
> > and
> > > > > jobs.
> > > > > > > > Those
> > > > > > > > > >> > > > interfaces
> > > > > > > > > >> > > > > > > will
> > > > > > > > > >> > > > > > > > > > define
> > > > > > > > > >> > > > > > > > > > > > > > methods
> > > > > > > > > >> > > > > > > > > > > > > > > like addNode, addVariable,
> > > > > updateTask,
> > > > > > > > > >> > > > > addAttachment.....
> > > > > > > > > >> > > > > > > > that
> > > > > > > > > >> > > > > > > > > > will
> > > > > > > > > >> > > > > > > > > > > > allow
> > > > > > > > > >> > > > > > > > > > > > > > > the persistent layer
> > > > implementation
> > > > > > to
> > > > > > > > just
> > > > > > > > > >> > update
> > > > > > > > > >> > > > the
> > > > > > > > > >> > > > > > > > needed
> > > > > > > > > >> > > > > > > > > > info
> > > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > DB  (for example, for
> addNode
> > in
> > > > > > > Postgres,
> > > > > > > > > >> just
> > > > > > > > > >> > > > insert
> > > > > > > > > >> > > > > a
> > > > > > > > > >> > > > > > > row
> > > > > > > > > >> > > > > > > > > into
> > > > > > > > > >> > > > > > > > > > > > nodes
> > > > > > > > > >> > > > > > > > > > > > > > > table, for addNode in Mongo,
> > > > > basically
> > > > > > > the
> > > > > > > > > >> same
> > > > > > > > > >> > > > atomic
> > > > > > > > > >> > > > > > > upsert
> > > > > > > > > >> > > > > > > > > > > > operation
> > > > > > > > > >> > > > > > > > > > > > > > > that is currently done).
> > > > Therefore,
> > > > > we
> > > > > > > > > >> increase
> > > > > > > > > >> > > > > > performance
> > > > > > > > > >> > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > Postgres
> > > > > > > > > >> > > > > > > > > > > > > > > and keep the current one for
> > > > Mongo.
> > > > > > The
> > > > > > > > > >> current
> > > > > > > > > >> > DB
> > > > > > > > > >> > > > > > schemas
> > > > > > > > > >> > > > > > > > > won't
> > > > > > > > > >> > > > > > > > > > be
> > > > > > > > > >> > > > > > > > > > > > > > > touched.
> > > > > > > > > >> > > > > > > > > > > > > > > Since the code change is
> > large,
> > > I
> > > > do
> > > > > > not
> > > > > > > > > think
> > > > > > > > > >> > I'll
> > > > > > > > > >> > > > be
> > > > > > > > > >> > > > > > able
> > > > > > > > > >> > > > > > > > to
> > > > > > > > > >> > > > > > > > > > have
> > > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > PR
> > > > > > > > > >> > > > > > > > > > > > > > > ready till next week.
> > > > > > > > > >> > > > > > > > > > > > > > > But before starting, please
> > let
> > > me
> > > > > > know
> > > > > > > if
> > > > > > > > > >> that
> > > > > > > > > >> > > > > approach
> > > > > > > > > >> > > > > > is
> > > > > > > > > >> > > > > > > > > fine
> > > > > > > > > >> > > > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > you.
> > > > > > > > > >> > > > > > > > > > > > > > > Best regards.
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> > 6:55 PM
> > > > Alex
> > > > > > > > > Porcelli
> > > > > > > > > >> <
> > > > > > > > > >> > > > > > > > > alex@porcelli.me>
> > > > > > > > > >> > > > > > > > > > > > wrote:
> > > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > Thank you Francisco to
> > getting
> > > > > > deeper
> > > > > > > on
> > > > > > > > > >> this…
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > Looking forward to see the
> > > > results
> > > > > > of
> > > > > > > > your
> > > > > > > > > >> > > > suggested
> > > > > > > > > >> > > > > > > > > > improvements.
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> > > 9:40 AM
> > > > > > > > Francisco
> > > > > > > > > >> > Javier
> > > > > > > > > >> > > > > Tirado
> > > > > > > > > >> > > > > > > > > Sarti <
> > > > > > > > > >> > > > > > > > > > > > > > > > ftirados@redhat.com>
> wrote:
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > I forgot to attach the
> > > queries
> > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> > > > 3:04 PM
> > > > > > > > > Francisco
> > > > > > > > > >> > > Javier
> > > > > > > > > >> > > > > > Tirado
> > > > > > > > > >> > > > > > > > > > Sarti <
> > > > > > > > > >> > > > > > > > > > > > > > > > > ftirados@redhat.com>
> > wrote:
> > > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > > > >> Hi,
> > > > > > > > > >> > > > > > > > > > > > > > > > >> A brief update on this
> > > topic.
> > > > > > > > > >> > > > > > > > > > > > > > > > >> After doing a simple
> test
> > > > with
> > > > > > > > example
> > > > > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > > > >> > > > > > > > > > > > > > > > ,
> > > > > > > > > >> > > > > > > > > > > > > > > > >> the number of updates
> > over
> > > > > Nodes
> > > > > > > > table
> > > > > > > > > is
> > > > > > > > > >> > n*n,
> > > > > > > > > >> > > > so
> > > > > > > > > >> > > > > we
> > > > > > > > > >> > > > > > > > > manage
> > > > > > > > > >> > > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > obtain a
> > > > > > > > > >> > > > > > > > > > > > > > > > >> perfect quadratic
> > > performance
> > > > > > > > > >> degradation.
> > > > > > > > > >> > The
> > > > > > > > > >> > > > > > problem
> > > > > > > > > >> > > > > > > > is
> > > > > > > > > >> > > > > > > > > > worse
> > > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > case
> > > > > > > > > >> > > > > > > > > > > > > > > > >> of Serverless Workflow
> > than
> > > > in
> > > > > > BPMN
> > > > > > > > > >> because
> > > > > > > > > >> > we
> > > > > > > > > >> > > > the
> > > > > > > > > >> > > > > > > > number
> > > > > > > > > >> > > > > > > > > of
> > > > > > > > > >> > > > > > > > > > > > nodes
> > > > > > > > > >> > > > > > > > > > > > > > is
> > > > > > > > > >> > > > > > > > > > > > > > > > >> greater than the number
> > of
> > > > > > states.
> > > > > > > In
> > > > > > > > > >> that
> > > > > > > > > >> > > > > example N
> > > > > > > > > >> > > > > > > is
> > > > > > > > > >> > > > > > > > > 16,
> > > > > > > > > >> > > > > > > > > > but
> > > > > > > > > >> > > > > > > > > > > > for
> > > > > > > > > >> > > > > > > > > > > > > > a
> > > > > > > > > >> > > > > > > > > > > > > > > > more
> > > > > > > > > >> > > > > > > > > > > > > > > > >> complex workflow it
> would
> > > be
> > > > > > > > certainly
> > > > > > > > > >> > large.
> > > > > > > > > >> > > > > > > > > > > > > > > > >> I think that this is
> more
> > > > > related
> > > > > > > to
> > > > > > > > > how
> > > > > > > > > >> we
> > > > > > > > > >> > > are
> > > > > > > > > >> > > > > > > handling
> > > > > > > > > >> > > > > > > > > > JPA in
> > > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > > > code,
> > > > > > > > > >> > > > > > > > > > > > > > > > >> in particular the
> mapping
> > > > from
> > > > > > > model
> > > > > > > > to
> > > > > > > > > >> > entity
> > > > > > > > > >> > > > > > > > (basically
> > > > > > > > > >> > > > > > > > > > JPA is
> > > > > > > > > >> > > > > > > > > > > > > > blind
> > > > > > > > > >> > > > > > > > > > > > > > > > and
> > > > > > > > > >> > > > > > > > > > > > > > > > >> has to update all nodes
> > for
> > > > > every
> > > > > > > > write
> > > > > > > > > >> > > because
> > > > > > > > > >> > > > it
> > > > > > > > > >> > > > > > > > > believes
> > > > > > > > > >> > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > node has
> > > > > > > > > >> > > > > > > > > > > > > > > > >> been updated, although
> it
> > > is
> > > > > not)
> > > > > > > > than
> > > > > > > > > an
> > > > > > > > > >> > > issue
> > > > > > > > > >> > > > in
> > > > > > > > > >> > > > > > the
> > > > > > > > > >> > > > > > > > > table
> > > > > > > > > >> > > > > > > > > > > > > > definition.
> > > > > > > > > >> > > > > > > > > > > > > > > > >> In fact, when using
> JPA,
> > > > > > separating
> > > > > > > > the
> > > > > > > > > >> > server
> > > > > > > > > >> > > > > model
> > > > > > > > > >> > > > > > > > from
> > > > > > > > > >> > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > JPA
> > > > > > > > > >> > > > > > > > > > > > > > > > entity is
> > > > > > > > > >> > > > > > > > > > > > > > > > >> not a good idea,
> > especially
> > > > if
> > > > > > the
> > > > > > > > > entity
> > > > > > > > > >> > > > contains
> > > > > > > > > >> > > > > > > > > > collections.
> > > > > > > > > >> > > > > > > > > > > > I
> > > > > > > > > >> > > > > > > > > > > > > > will
> > > > > > > > > >> > > > > > > > > > > > > > > > try
> > > > > > > > > >> > > > > > > > > > > > > > > > >> to change that without
> > > > breaking
> > > > > > > > > anything.
> > > > > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > > > > >> > > > > > > > > > > > > > > > >> On Wed, Nov 22, 2023 at
> > > > > 12:10 PM
> > > > > > > > > Enrique
> > > > > > > > > >> > > > Gonzalez
> > > > > > > > > >> > > > > > > > > Martinez <
> > > > > > > > > >> > > > > > > > > > > > > > > > >> egonzalez@apache.org>
> > > wrote:
> > > > > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> After the events split
> > you
> > > > now
> > > > > > > will
> > > > > > > > > >> need to
> > > > > > > > > >> > > > > create
> > > > > > > > > >> > > > > > a
> > > > > > > > > >> > > > > > > > node
> > > > > > > > > >> > > > > > > > > > > > instance
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> model instance of
> making
> > > > > > > independent
> > > > > > > > > >> from
> > > > > > > > > >> > the
> > > > > > > > > >> > > > > > process
> > > > > > > > > >> > > > > > > > > > instance.
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> That should do the
> > trick.
> > > > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> Regarding
> > > deleting/inserting
> > > > > it
> > > > > > > was
> > > > > > > > > >> fixed
> > > > > > > > > >> > at
> > > > > > > > > >> > > > some
> > > > > > > > > >> > > > > > > > point.
> > > > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> El mar, 21 nov 2023 a
> > las
> > > > > 20:22,
> > > > > > > > > >> Francisco
> > > > > > > > > >> > > > Javier
> > > > > > > > > >> > > > > > > > Tirado
> > > > > > > > > >> > > > > > > > > > Sarti
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> (<ftirados@redhat.com
> >)
> > > > > > escribió:
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > Hi Martin,
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > I have a task to
> > review
> > > > > > > > performance
> > > > > > > > > of
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > ProcessInstanceNodeDataEventMerger
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > My idea is to reduce
> > the
> > > > > > number
> > > > > > > of
> > > > > > > > > >> delete
> > > > > > > > > >> > > > > inserts
> > > > > > > > > >> > > > > > > > when
> > > > > > > > > >> > > > > > > > > > > > processing
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> events
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > and try to do it
> > > > > incremental.
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > That should improve
> > > > > > performance.
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > PS:
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > I was planning to
> send
> > > an
> > > > > > e-mail
> > > > > > > > > >> tomorrow
> > > > > > > > > >> > > > > > > announcing
> > > > > > > > > >> > > > > > > > > > that in
> > > > > > > > > >> > > > > > > > > > > > > > case you
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> were
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > already working on a
> > fix
> > > > for
> > > > > > > > that. I
> > > > > > > > > >> > assume
> > > > > > > > > >> > > > you
> > > > > > > > > >> > > > > > are
> > > > > > > > > >> > > > > > > > not
> > > > > > > > > >> > > > > > > > > > and I
> > > > > > > > > >> > > > > > > > > > > > > > would
> > > > > > > > > >> > > > > > > > > > > > > > > > be
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > sending a PR soon.
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > >> > > > > > > > > > > > > > > > >>> > On Tue, Nov 21, 2023
> > at
> > > > > > 6:09 PM
> > > > > > > > > Martin
> > > > > > > > > >> > > Weiler
> > > > > > > > > >> > > > > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > >> > > > > > > > > > > > > > > > >>
>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Richard Bourner <ri...@gmail.com>]

Got it. Thanks Enrique.


Le mer. 21 févr. 2024, 00:18, Enrique Gonzalez Martinez <
egonzalez@apache.org> a écrit :

> Regarding your STP concerns, they were addressed in my previous comment:
>
> STP is a concept, a process with certain constraints: no persistence and
> returning the outcome in the call (sync execution with no idle states). It
> was a requirement from a user in the past. One of the requirements was
> leaving no trail. In v7 was easy because you could disable the audit in
> that case. Actually we have the same way to do what we did in v7 in here as
> you can add/remove index just removing dependencies.
>
> So in fact in microservice you just need to exclude the data index from the
> app and you wont have data index.
>
>
>
> El mié, 21 feb 2024, 1:40, Richard Bourner <ri...@gmail.com> escribió:
>
> > +1 with Martin's email.
> >
> > One question though in regards to Martin's point #3 and to previous
> > following statement from Francisco: "*... keeping finishing
> > process instances "for a while" in DataIndex was the only way for users
> to
> > query the result of straight through processes*"
> > --> Is this the only use case where data index would be needed for STP?
> > I am asking because clients will already get their result in the JSON
> > returned from the synchronous REST call, so adding an extra computing
> time
> > for data index persistence does not seem right to me in the context of
> > decision services that are supposed to be very fast to return (typically
> > rule tasks+scripts).
> > Or is it that we also want to provide some GraphQL capabilities, even for
> > STP use cases?
> >
> > Also, what would "*for a while*" mean exactly?  Will it be configurable?
> > Will there be a default expiration value?
> >
> > I am assuming this is all work in progress, and you may not have answers
> to
> > all my questions, no problem with that.
> >
> > Thanks.
> >
> > On Tue, Feb 20, 2024 at 5:39 PM Martin Weiler <mw...@ibm.com.invalid>
> > wrote:
> >
> > > IMO, it is good to have this discussion around data sanity now instead
> of
> > > putting it off until later when data has already accumulated in
> > production
> > > environments.
> > >
> > > Based on the input here, we are dealing with three types of data:
> > > 1. Runtime data - active instances only, engine cleans up the data
> > > automatically at process instance end
> > > 2. Historic log data - data created by data-audit intended for long
> term
> > > storage
> > > 3. Data-index data - somehow this data falls in between the two
> > > aforementioned categories, with the idea of the data being "recent",
> but
> > > not restricted to active instances only
> > >
> > > We'd need purge strategies for both #2 and #3 (perhaps different ones,
> or
> > > with different config settings) in order to prevent unlimited data
> > growth.
> > >
> > > ________________________________________
> > > From: Enrique Gonzalez Martinez <eg...@apache.org>
> > > Sent: Monday, February 19, 2024 7:11 AM
> > > To: dev@kie.apache.org
> > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with data-index
> > > persistence addon
> > >
> > > Hi Francisco,
> > > To give you more context about this.
> > >
> > > STP is a concept, a process with certain constraints: no persistence
> and
> > > returning the outcome in the call (sync execution with no idle states).
> > It
> > > was a requirement from a user in the past. One of the requirements was
> > > leaving no trail. In v7 was easy because you could disable the audit in
> > > that case. Actually we have the same way to do what we did in v7 in
> here
> > as
> > > you can add/remove index just removing deps.
> > >
> > > We have the same outcome with different approaches and STP is already
> > > delivered.
> > >
> > > El lun, 19 feb 2024 a las 14:46, Francisco Javier Tirado Sarti (<
> > > ftirados@redhat.com>) escribió:
> > >
> > > > Regarding STP (which is not a concept that we have in the code. I
> mean
> > > STP
> > > > are processes as nonSTP are), I guess, as all processes, they were
> kept
> > > in
> > > > DataIndex once completed because users wanted (and still wants) to
> > check
> > > > the result once the call had been performed. If we want to leave no
> > trace
> > > > of them in DataIndex for some reason, we will need to make it a
> > > > Runtimes concept so DataIndex can handle them in a different way.
> > > >
> > > > On Mon, Feb 19, 2024 at 2:27 PM Enrique Gonzalez Martinez <
> > > > egonzalez@apache.org> wrote:
> > > >
> > > > > Alex:
> > > > > Right now the data index is working in the same way as it did in v7
> > > with
> > > > > the emitters. The only difference between two impl is that in here
> > the
> > > > > storage is pgsql instead elastic search.  You are right regarding
> is
> > a
> > > > > snapshot of the last state of the process but we did never define
> how
> > > > long
> > > > > would be alive that dats Honestly i am happy right now with the way
> > it
> > > > > works. The clean up mechanism is still tbd because we still need to
> > > > discuss
> > > > > other stuff first.
> > > > >
> > > > >
> > > > > Regarding stp is to leave no trail because u can get the outcome
> > > directly
> > > > > from the call. It was defined like that in v7. So there is no use
> for
> > > the
> > > > > index or the audit.
> > > > >
> > > > > El lun, 19 feb 2024, 14:13, Francisco Javier Tirado Sarti <
> > > > > ftirados@redhat.com> escribió:
> > > > >
> > > > > > Hi Alex,
> > > > > > There has been some confusion about the purpose of DataIndex. To
> be
> > > > > honest
> > > > > > I believe they were already sorted out, but your e-mail makes me
> > > think
> > > > > that
> > > > > > is not the case ;). I let Kris to clarify that with you. My view
> is
> > > > that
> > > > > > data-index is a way to query recently closed and active processes
> > > (the
> > > > > key
> > > > > > here is the definition of recently, which in my opinion should be
> > > > > > configurable)
> > > > > > But, besides that discussion and being pragmatic, keeping
> finishing
> > > > > process
> > > > > > instances "for a while" in DataIndex was the only way for users
> to
> > > > query
> > > > > > the result of straight through processes. That's a function that
> > > cannot
> > > > > be
> > > > > > removed right now
> > > > > >
> > > > > > On Mon, Feb 19, 2024 at 1:33 PM Alex Porcelli <
> porcelli@apache.org
> > >
> > > > > wrote:
> > > > > >
> > > > > > > if data index was supposed to provide snapshot view of the
> > process
> > > > > > > instance… why do we keep it after the process instance is
> > finished?
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Feb 19, 2024 at 7:12 AM Francisco Javier Tirado Sarti <
> > > > > > > ftirados@redhat.com> wrote:
> > > > > > >
> > > > > > > > Hi Martin.
> > > > > > > > After taking a deeper look at this, I realize that the
> > behaviour
> > > is
> > > > > the
> > > > > > > > expected one.
> > > > > > > > Runtimes DB does not track the completed process instance
> > (that's
> > > > > what
> > > > > > > the
> > > > > > > > JDBCProcessInstances warn is telling us), but DataIndex, as
> > > > expected,
> > > > > > is
> > > > > > > > tracking it in processes and nodes table. And yes it will
> grow
> > > over
> > > > > > time.
> > > > > > > > What we need is some configurable purge mechanism for
> > DataIndex,
> > > so
> > > > > it
> > > > > > > > eventually removes older completed process instances.
> > > > > > > >
> > > > > > > > On Tue, Feb 13, 2024 at 12:59 PM Francisco Javier Tirado
> Sarti
> > <
> > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > >
> > > > > > > > > Hi Martin,
> > > > > > > > > Good catch!. Looks like the skipping performed for process
> > > > > instances
> > > > > > is
> > > > > > > > > not applied to node instances. Something we definitely need
> > to
> > > > > review
> > > > > > > on
> > > > > > > > > the runtimes side.
> > > > > > > > >
> > > > > > > > > On Mon, Feb 12, 2024 at 11:59 PM Martin Weiler
> > > > > > <mweiler@ibm.com.invalid
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> On a somewhat related note, testing a simple workflow
> (start
> > > ->
> > > > > > script
> > > > > > > > >> node -> end), I see the following messages in the logs:
> > > > > > > > >> 2024-02-12 22:49:50,493 28758dde544c WARN
> > > > > > > > >> [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> > > > > > > > >> (executor-thread-3) Skipping create of process instance
> id:
> > > > > > > > >> 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2
> > > > > > > > >>
> > > > > > > > >> So far, so good. And I'd expect to see no trace of this
> > > process
> > > > in
> > > > > > the
> > > > > > > > >> database if I don't have data audit enabled.
> > > > > > > > >>
> > > > > > > > >> However, the 'processes' table contains a row with
> state=2,
> > > with
> > > > > > > related
> > > > > > > > >> entries in the 'nodes' table. In a load test, I see these
> > > tables
> > > > > > grow
> > > > > > > > >> significantly over time. Am I missing something to have
> > these
> > > > > > entries
> > > > > > > > >> cleaned up automatically?
> > > > > > > > >>
> > > > > > > > >> ________________________________________
> > > > > > > > >> From: Martin Weiler <mw...@ibm.com.INVALID>
> > > > > > > > >> Sent: Monday, February 12, 2024 3:40 PM
> > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > >> Subject: [EXTERNAL] RE: [DISCUSSION] Performance issues
> with
> > > > > > > data-index
> > > > > > > > >> persistence addon
> > > > > > > > >>
> > > > > > > > >> Thanks everyone for your input. Based on this discussion,
> I
> > > > opened
> > > > > > the
> > > > > > > > >> following PR:
> > > > > > > > >>
> > https://github.com/apache/incubator-kie-kogito-apps/pull/1985
> > > > > > > > >>
> > > > > > > > >> With this change, the performance seems to be stable over
> > > time:
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
> > > > > > > > >>
> > > > > > > > >> Martin
> > > > > > > > >>
> > > > > > > > >> ________________________________________
> > > > > > > > >> From: Gonzalo Muñoz <gm...@apache.org>
> > > > > > > > >> Sent: Friday, February 9, 2024 9:42 AM
> > > > > > > > >> To: dev@kie.apache.org
> > > > > > > > >> Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues
> with
> > > > > > > data-index
> > > > > > > > >> persistence addon
> > > > > > > > >>
> > > > > > > > >> Great work Francisco,
> > > > > > > > >> Martin, take a look at this link with some related tips
> (in
> > > case
> > > > > you
> > > > > > > > find
> > > > > > > > >> it useful):
> > > > > > > > >>
> > > https://www.cybertec-postgresql.com/en/index-your-foreign-key/
> > > > > > > > >>
> > > > > > > > >> El vie, 9 feb 2024 a las 17:20, Francisco Javier Tirado
> > Sarti
> > > (<
> > > > > > > > >> ftirados@redhat.com>) escribió:
> > > > > > > > >>
> > > > > > > > >> > For the moment being, we will keep JPA till we exhaust
> all
> > > > > > > > >> possibilities,
> > > > > > > > >> > let's call switching from jpa to jdbc our hidden plan B
> ;)
> > > > > > > > >> > I already told Martin, but in order everyone to know,
> just
> > > > after
> > > > > > > > writing
> > > > > > > > >> > the previous email, I thought "what if Postgres is not
> > > > > > automatically
> > > > > > > > >> > indexing foreign keys like mysql?" and, eureka
> > > > > > > > >> > Postgres doc
> > > > > > > > >> >
> > > https://www.postgresql.org/docs/current/ddl-constraints.html
> > > > > > > > >> > Mysql doc
> > > > > > > > >> >
> > > > > >
> > https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> > > > > > > > >> > These are the relevant excerpt
> > > > > > > > >> >
> > > > > > > > >> > *Postgresql*
> > > > > > > > >> > *A foreign key must reference columns that either are a
> > > > primary
> > > > > > key
> > > > > > > or
> > > > > > > > >> form
> > > > > > > > >> > a unique constraint, or are columns from a non-partial
> > > unique
> > > > > > index.
> > > > > > > > >> This
> > > > > > > > >> > means that the referenced columns always have an index
> to
> > > > allow
> > > > > > > > >> efficient
> > > > > > > > >> > lookups on whether a referencing row has a match. Since
> a
> > > > DELETE
> > > > > > of
> > > > > > > a
> > > > > > > > >> row
> > > > > > > > >> > from the referenced table or an UPDATE of a referenced
> > > column
> > > > > will
> > > > > > > > >> require
> > > > > > > > >> > a scan of the referencing table for rows matching the
> old
> > > > value,
> > > > > > it
> > > > > > > is
> > > > > > > > >> > often a good idea to index the referencing columns too.
> > > > Because
> > > > > > this
> > > > > > > > is
> > > > > > > > >> not
> > > > > > > > >> > always needed, and there are many choices available on
> how
> > > to
> > > > > > index,
> > > > > > > > the
> > > > > > > > >> > declaration of a foreign key constraint does not
> > > automatically
> > > > > > > create
> > > > > > > > an
> > > > > > > > >> > index on the referencing columns.*
> > > > > > > > >> > *Mysql*
> > > > > > > > >> > *MySQL requires that foreign key columns be indexed; if
> > you
> > > > > > create a
> > > > > > > > >> table
> > > > > > > > >> > with a foreign key constraint but no index on a given
> > > column,
> > > > an
> > > > > > > index
> > > > > > > > >> is
> > > > > > > > >> > created. *
> > > > > > > > >> >
> > > > > > > > >> > So I asked Martin to especially create an index for
> > > > > > > > process_instance_id
> > > > > > > > >> > column on nodes table
> > > > > > > > >> > I think that will fix the problem detected on the thread
> > > dump.
> > > > > > > > >> > The simpler process test to verify queries are fine
> still
> > > > > stands,
> > > > > > > > >> though ;)
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <
> > > > > tzimanyi@apache.org
> > > > > > >
> > > > > > > > >> wrote:
> > > > > > > > >> >
> > > > > > > > >> > > I always preferred pure JDBC over Hibernate myself,
> just
> > > for
> > > > > the
> > > > > > > > sake
> > > > > > > > >> of
> > > > > > > > >> > > control of what is happening :) So I would not -1 that
> > > > myself.
> > > > > > > > >> > >
> > > > > > > > >> > > Tibor
> > > > > > > > >> > >
> > > > > > > > >> > > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado
> Sarti <
> > > > > > > > >> > > ftirados@redhat.com>
> > > > > > > > >> > > napísal(a):
> > > > > > > > >> > >
> > > > > > > > >> > > > Hi,
> > > > > > > > >> > > > Usually I do not want to talk about work in progress
> > > > because
> > > > > > > > >> > preliminary
> > > > > > > > >> > > > conclusions are pretty volatile but, well, there
> are a
> > > > > couple
> > > > > > of
> > > > > > > > >> things
> > > > > > > > >> > > > that can be concluded from the really valuable
> > > information
> > > > > > that
> > > > > > > > >> Martin
> > > > > > > > >> > > > provided.
> > > > > > > > >> > > > 1) In order to be able to determine if the number of
> > > > > > statements
> > > > > > > is
> > > > > > > > >> > larger
> > > > > > > > >> > > > than expected, I asked Martin to test with a simpler
> > > > process
> > > > > > > > >> > definition.
> > > > > > > > >> > > > One with just three nodes: start, script and end.
> The
> > > > script
> > > > > > one
> > > > > > > > >> should
> > > > > > > > >> > > > change just one variable. This way we can analyze if
> > the
> > > > > > number
> > > > > > > of
> > > > > > > > >> > > queries
> > > > > > > > >> > > > is the expected one. From the single log (audit was
> > > > > activated
> > > > > > > > them)
> > > > > > > > >> my
> > > > > > > > >> > > > conclusion is that the number of insert/updates over
> > > > > processes
> > > > > > > and
> > > > > > > > >> > nodes
> > > > > > > > >> > > > (there a lot over task, that I will prefer to skip
> for
> > > > now,
> > > > > > baby
> > > > > > > > >> steps)
> > > > > > > > >> > > is
> > > > > > > > >> > > > the expected one.
> > > > > > > > >> > > > 2) Analysing the thread dump, we see around 15
> threads
> > > > > > executing
> > > > > > > > >> this
> > > > > > > > >> > > line
> > > > > > > > >> > > > at
> > > > > > > > >> > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > > > > > > >> > > > so its pretty clear the code to be optimized ;). I'm
> > > > > > evaluating
> > > > > > > > >> > > > possibilities within JPA/Hibernate, but I'm starting
> > to
> > > > > think
> > > > > > > that
> > > > > > > > >> it
> > > > > > > > >> > > might
> > > > > > > > >> > > > be better to switch to JDBC and skip hibernate. Our
> > > lives
> > > > > will
> > > > > > > be
> > > > > > > > >> > > simpler,
> > > > > > > > >> > > > especially with a schema relatively simple like ours
> > > (that
> > > > > > will
> > > > > > > be
> > > > > > > > >> my
> > > > > > > > >> > > > recommendation if I was an external consultant)
> > > > > > > > >> > > >
> > > > > > > > >> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <
> > > > > > > tzimanyi@apache.org
> > > > > > > > >
> > > > > > > > >> > > wrote:
> > > > > > > > >> > > >
> > > > > > > > >> > > > > Hi,
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > this will be a bit off-topic. However as far as
> > > > > > performance, I
> > > > > > > > >> think
> > > > > > > > >> > we
> > > > > > > > >> > > > > should think about that we have string primary
> keys
> > > > > (IDs). I
> > > > > > > > would
> > > > > > > > >> > > expect
> > > > > > > > >> > > > > the database systems are much better with indexing
> > > > numeric
> > > > > > > keys
> > > > > > > > >> than
> > > > > > > > >> > > > > strings. I remember from the past, when I was
> > working
> > > > with
> > > > > > > DBs,
> > > > > > > > >> that
> > > > > > > > >> > > > using
> > > > > > > > >> > > > > strings as keys or indexes was a discouraged
> > practice.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Best regards,
> > > > > > > > >> > > > > Tibor
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler
> > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > >
> > > > > > > > >> > > > > napísal(a):
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > > I changed the test to use MongoDB [1] and I
> don't
> > > see
> > > > a
> > > > > > > > >> performance
> > > > > > > > >> > > > > > degradation with this setup [2].
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > Please keep us posted of your findings. Thanks!
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > Martin
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > [1]
> > > > > > > > >> > > > >
> > > > > > > > >> >
> > > > > > > >
> > > > >
> > https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > > > > > >> > > > > > [2]
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > ________________________________________
> > > > > > > > >> > > > > > From: Francisco Javier Tirado Sarti <
> > > > > ftirados@redhat.com>
> > > > > > > > >> > > > > > Sent: Wednesday, February 7, 2024 11:40 AM
> > > > > > > > >> > > > > > To: dev@kie.apache.org
> > > > > > > > >> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance
> > > > issues
> > > > > > with
> > > > > > > > >> > > data-index
> > > > > > > > >> > > > > > persistence addon
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > yes, it can be index degradation because of
> size,
> > > but
> > > > I
> > > > > > > > believe
> > > > > > > > >> (I
> > > > > > > > >> > > > might
> > > > > > > > >> > > > > be
> > > > > > > > >> > > > > > wrong) the db is too small (yet) for that.
> > > > > > > > >> > > > > > But, eventually, Postgres, when the DB is huge
> > > enough,
> > > > > > > > >> unavoidably
> > > > > > > > >> > > will
> > > > > > > > >> > > > > > behave like the graphic that Martin sent.
> > > > > > > > >> > > > > > Since I believe we are not huge enough (yet),
> lets
> > > > rule
> > > > > > out
> > > > > > > > >> another
> > > > > > > > >> > > > issue
> > > > > > > > >> > > > > > by analysing the sql logs (I requested those to
> > > Martin
> > > > > > > offline
> > > > > > > > >> and
> > > > > > > > >> > he
> > > > > > > > >> > > > is
> > > > > > > > >> > > > > > going to kindly collect them).
> > > > > > > > >> > > > > > Also Im curious to know if Mongo behave in the
> > same
> > > > way.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez
> > > > > Martinez <
> > > > > > > > >> > > > > > egonzalez@apache.org> wrote:
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > > Hi Francisco,
> > > > > > > > >> > > > > > > I would highly recommend to check indexes and
> > how
> > > > the
> > > > > > > > updates
> > > > > > > > >> > work
> > > > > > > > >> > > in
> > > > > > > > >> > > > > > data
> > > > > > > > >> > > > > > > index to avoid full scan table and lock the
> full
> > > > > table.
> > > > > > > Some
> > > > > > > > >> db
> > > > > > > > >> > are
> > > > > > > > >> > > > > very
> > > > > > > > >> > > > > > > sensitive to that.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > El mié, 7 feb 2024, 18:41, Francisco Javier
> > Tirado
> > > > > > Sarti <
> > > > > > > > >> > > > > > > ftirados@redhat.com> escribió:
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > > Hi Martin,
> > > > > > > > >> > > > > > > > While I analyze the data, let me ask you if
> it
> > > is
> > > > > > > possible
> > > > > > > > >> to
> > > > > > > > >> > > > perform
> > > > > > > > >> > > > > > > > another check (similar in a way to disabling
> > > > > > data-index
> > > > > > > > like
> > > > > > > > >> > you
> > > > > > > > >> > > > do)
> > > > > > > > >> > > > > > Can
> > > > > > > > >> > > > > > > > you switch to MongoDB persistence and check
> if
> > > the
> > > > > > same
> > > > > > > > >> > > degradation
> > > > > > > > >> > > > > > that
> > > > > > > > >> > > > > > > is
> > > > > > > > >> > > > > > > > there for postgres remains?
> > > > > > > > >> > > > > > > > I do not know if this is feasible but will
> > > > certainly
> > > > > > > > >> indicate
> > > > > > > > >> > the
> > > > > > > > >> > > > > > problem
> > > > > > > > >> > > > > > > > is on the postgres storage layer and I do
> not
> > > > have a
> > > > > > > clear
> > > > > > > > >> > > > prediction
> > > > > > > > >> > > > > > of
> > > > > > > > >> > > > > > > > what we will see when doing this switch.
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler
> > > > > > > > >> > > > <mweiler@ibm.com.invalid
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > > > wrote:
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > > Hi Francisco,
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > thanks for your work on this important
> > topic!
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > I would like to share some test results
> > here,
> > > > > which
> > > > > > > > might
> > > > > > > > >> > help
> > > > > > > > >> > > to
> > > > > > > > >> > > > > > > improve
> > > > > > > > >> > > > > > > > > the codebase even further. I am using the
> > > jmeter
> > > > > > based
> > > > > > > > >> test
> > > > > > > > >> > > case
> > > > > > > > >> > > > > from
> > > > > > > > >> > > > > > > > Pere
> > > > > > > > >> > > > > > > > > and Enrique (thanks guys!) [1] which uses
> a
> > > load
> > > > > of
> > > > > > 30
> > > > > > > > >> > threads
> > > > > > > > >> > > to
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > 1) start a new process instance (POST)
> > > > > > > > >> > > > > > > > > 2) retrieve tasks for a user (GET)
> > > > > > > > >> > > > > > > > > 3) fetches task details (GET)
> > > > > > > > >> > > > > > > > > 4) complete a task (POST)
> > > > > > > > >> > > > > > > > > 5) execute a query on data-audit
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > With this test setup, I noticed that the
> > > > > performance
> > > > > > > for
> > > > > > > > >> the
> > > > > > > > >> > > POST
> > > > > > > > >> > > > > > > > > requests, in particular the one to start a
> > new
> > > > > > process
> > > > > > > > >> > > instance,
> > > > > > > > >> > > > > > > degrades
> > > > > > > > >> > > > > > > > > over time - see graph [2]. If I run the
> same
> > > > test
> > > > > > > > without
> > > > > > > > >> > > > > data-index,
> > > > > > > > >> > > > > > > > then
> > > > > > > > >> > > > > > > > > there is no such performance degradation
> > [3].
> > > > You
> > > > > > can
> > > > > > > > >> find a
> > > > > > > > >> > > > thread
> > > > > > > > >> > > > > > > dump
> > > > > > > > >> > > > > > > > > captured a few minutes into the first test
> > > here
> > > > > [4]
> > > > > > > that
> > > > > > > > >> > might
> > > > > > > > >> > > > help
> > > > > > > > >> > > > > > to
> > > > > > > > >> > > > > > > > see
> > > > > > > > >> > > > > > > > > some of the contention points.
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > I'd appreciate if you could take a look
> and
> > > see
> > > > if
> > > > > > > there
> > > > > > > > >> is
> > > > > > > > >> > > > > something
> > > > > > > > >> > > > > > > > that
> > > > > > > > >> > > > > > > > > can be further improved based on your
> > previous
> > > > > work.
> > > > > > > If
> > > > > > > > >> you
> > > > > > > > >> > > need
> > > > > > > > >> > > > > any
> > > > > > > > >> > > > > > > > > additional data, let me know, but
> otherwise
> > it
> > > > is
> > > > > > > > >> > > straightforward
> > > > > > > > >> > > > > to
> > > > > > > > >> > > > > > > run
> > > > > > > > >> > > > > > > > > the jmeter test as well.
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > Thanks,
> > > > > > > > >> > > > > > > > > Martin
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > [1]
> > > > > > > > >> https://github.com/pefernan/job-service-refactor-test/
> > > > > > > > >> > > > > > > > > [2]
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > > > > >> > > > > > > > > [3]
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > > > > >> > > > > > > > > [4]
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > ________________________________________
> > > > > > > > >> > > > > > > > > From: Francisco Javier Tirado Sarti <
> > > > > > > > ftirados@redhat.com>
> > > > > > > > >> > > > > > > > > Sent: Wednesday, January 17, 2024 9:13 AM
> > > > > > > > >> > > > > > > > > To: dev@kie.apache.org
> > > > > > > > >> > > > > > > > > Cc: Pere Fernandez Perez
> > > > > > > > >> > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION]
> > > Performance
> > > > > > > issues
> > > > > > > > >> with
> > > > > > > > >> > > > > > data-index
> > > > > > > > >> > > > > > > > > persistence addon
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > Hi Alex,
> > > > > > > > >> > > > > > > > > I did not take times (which depends on a
> > > number
> > > > of
> > > > > > > > >> variables
> > > > > > > > >> > > that
> > > > > > > > >> > > > > > > > > drastically change between environments),
> > but
> > > > > verify
> > > > > > > > that
> > > > > > > > >> the
> > > > > > > > >> > > > > number
> > > > > > > > >> > > > > > of
> > > > > > > > >> > > > > > > > > updates has been reduced drastically
> without
> > > > > losing
> > > > > > > > >> > > > functionality,
> > > > > > > > >> > > > > > > which
> > > > > > > > >> > > > > > > > is
> > > > > > > > >> > > > > > > > > objectively a good thing. If before the
> > > change,
> > > > > for
> > > > > > > > every
> > > > > > > > >> > node
> > > > > > > > >> > > > > > > executed,
> > > > > > > > >> > > > > > > > we
> > > > > > > > >> > > > > > > > > have an update for every node previously
> > > > executed,
> > > > > > so
> > > > > > > > if a
> > > > > > > > >> > > > process
> > > > > > > > >> > > > > > have
> > > > > > > > >> > > > > > > > 50
> > > > > > > > >> > > > > > > > > nodes to execute, we were performing
> nearly
> > > > > 50*51/2
> > > > > > > > >> updates,
> > > > > > > > >> > > > which
> > > > > > > > >> > > > > > > gives
> > > > > > > > >> > > > > > > > us
> > > > > > > > >> > > > > > > > > a total of  1275 updates, now we have just
> > one
> > > > for
> > > > > > > every
> > > > > > > > >> node
> > > > > > > > >> > > > being
> > > > > > > > >> > > > > > > > > executed, implying a total of 50 updates.
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex
> > Porcelli
> > > <
> > > > > > > > >> > > alex@porcelli.me>
> > > > > > > > >> > > > > > > wrote:
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > > Francisco,
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > I noticed that your PR has been merged,
> > but
> > > I
> > > > > was
> > > > > > > > >> expecting
> > > > > > > > >> > > (at
> > > > > > > > >> > > > > > least
> > > > > > > > >> > > > > > > > > > was my understanding from this thread)
> > that
> > > > > before
> > > > > > > > >> merging
> > > > > > > > >> > > some
> > > > > > > > >> > > > > > > > > > benchmark data would be shared in
> advance
> > -
> > > to
> > > > > > > assess
> > > > > > > > >> the
> > > > > > > > >> > > > > > > cost/benefit
> > > > > > > > >> > > > > > > > > > of such a decent size change.
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > Do you have any information to share?
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM
> Francisco
> > > > Javier
> > > > > > > > Tirado
> > > > > > > > >> > Sarti
> > > > > > > > >> > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > Yes, as intended, now we have one
> select
> > > and
> > > > > one
> > > > > > > > >> > > > insert/update
> > > > > > > > >> > > > > > per
> > > > > > > > >> > > > > > > > node
> > > > > > > > >> > > > > > > > > > > event.
> > > > > > > > >> > > > > > > > > > > I moved the PR as ready for review and
> > > give
> > > > > > @Pere
> > > > > > > > >> > Fernandez
> > > > > > > > >> > > > > Perez
> > > > > > > > >> > > > > > > > > > > <pe...@redhat.com> permission to
> the
> > > > > branch
> > > > > > so
> > > > > > > > he
> > > > > > > > >> can
> > > > > > > > >> > > > edit
> > > > > > > > >> > > > > it
> > > > > > > > >> > > > > > > in
> > > > > > > > >> > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > next two weeks (Ill be on PTO)  if
> > > desired,
> > > > > > before
> > > > > > > > >> > merging.
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex
> > > > Porcelli
> > > > > <
> > > > > > > > >> > > > > alex@porcelli.me>
> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > Cool, thank you Francisco!
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > Did you manage to get some
> preliminary
> > > > data
> > > > > > > about
> > > > > > > > >> > > > > improvements?
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM
> > > Francisco
> > > > > > > Javier
> > > > > > > > >> > Tirado
> > > > > > > > >> > > > > Sarti
> > > > > > > > >> > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > Yes, after some delay because of
> > > > quarkus 3
> > > > > > > > >> migration.
> > > > > > > > >> > > Im
> > > > > > > > >> > > > > > > refining
> > > > > > > > >> > > > > > > > > > this
> > > > > > > > >> > > > > > > > > > > > > draft PR
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > On Thu, Dec 21, 2023 at 5:48 PM
> Alex
> > > > > > Porcelli
> > > > > > > <
> > > > > > > > >> > > > > > > alex@porcelli.me>
> > > > > > > > >> > > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > Any update or new findings on
> this
> > > > > topic?
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM
> > > > > Francisco
> > > > > > > > Javier
> > > > > > > > >> > > Tirado
> > > > > > > > >> > > > > > Sarti
> > > > > > > > >> > > > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > Hi Alex,
> > > > > > > > >> > > > > > > > > > > > > > > After considering different
> > > options
> > > > to
> > > > > > > > improve
> > > > > > > > >> > > > > > performance,
> > > > > > > > >> > > > > > > > we
> > > > > > > > >> > > > > > > > > > feel
> > > > > > > > >> > > > > > > > > > > > that
> > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > > >> > > > > > > > > > > > > > > is time to "partially" move
> away
> > > > from
> > > > > > the
> > > > > > > > >> current
> > > > > > > > >> > > Map
> > > > > > > > >> > > > > > style
> > > > > > > > >> > > > > > > > > > > > interface (
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > > > >> > > > > > > > > > > > > > )
> > > > > > > > >> > > > > > > > > > > > > > > which was shared with Trusty,
> to
> > > one
> > > > > > more
> > > > > > > > >> > suitable
> > > > > > > > >> > > > for
> > > > > > > > >> > > > > > > usage
> > > > > > > > >> > > > > > > > > > with a
> > > > > > > > >> > > > > > > > > > > > > > > relational DB like postgresql
> > (but
> > > > > still
> > > > > > > > >> > compatible
> > > > > > > > >> > > > > with
> > > > > > > > >> > > > > > > big
> > > > > > > > >> > > > > > > > > > table
> > > > > > > > >> > > > > > > > > > > > dbs).
> > > > > > > > >> > > > > > > > > > > > > > > The idea will be to replace
> > > generic
> > > > > > > Storage
> > > > > > > > >> > > interface
> > > > > > > > >> > > > > by
> > > > > > > > >> > > > > > > four
> > > > > > > > >> > > > > > > > > > > > specific
> > > > > > > > >> > > > > > > > > > > > > > > interfaces (which will inherit
> > > from
> > > > a
> > > > > > > common
> > > > > > > > >> one
> > > > > > > > >> > > that
> > > > > > > > >> > > > > > keeps
> > > > > > > > >> > > > > > > > the
> > > > > > > > >> > > > > > > > > > query
> > > > > > > > >> > > > > > > > > > > > > > part
> > > > > > > > >> > > > > > > > > > > > > > > at is it. with get and query
> > > > methods),
> > > > > > > that
> > > > > > > > >> will
> > > > > > > > >> > > > > include
> > > > > > > > >> > > > > > > the
> > > > > > > > >> > > > > > > > > > required
> > > > > > > > >> > > > > > > > > > > > > > > modification operations for
> the
> > > four
> > > > > > > > DataIndex
> > > > > > > > >> > > > > "domains":
> > > > > > > > >> > > > > > > > > > > > > > processinstance,
> > > > > > > > >> > > > > > > > > > > > > > > usertask, processdefinitions
> and
> > > > jobs.
> > > > > > > Those
> > > > > > > > >> > > > interfaces
> > > > > > > > >> > > > > > > will
> > > > > > > > >> > > > > > > > > > define
> > > > > > > > >> > > > > > > > > > > > > > methods
> > > > > > > > >> > > > > > > > > > > > > > > like addNode, addVariable,
> > > > updateTask,
> > > > > > > > >> > > > > addAttachment.....
> > > > > > > > >> > > > > > > > that
> > > > > > > > >> > > > > > > > > > will
> > > > > > > > >> > > > > > > > > > > > allow
> > > > > > > > >> > > > > > > > > > > > > > > the persistent layer
> > > implementation
> > > > > to
> > > > > > > just
> > > > > > > > >> > update
> > > > > > > > >> > > > the
> > > > > > > > >> > > > > > > > needed
> > > > > > > > >> > > > > > > > > > info
> > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > DB  (for example, for addNode
> in
> > > > > > Postgres,
> > > > > > > > >> just
> > > > > > > > >> > > > insert
> > > > > > > > >> > > > > a
> > > > > > > > >> > > > > > > row
> > > > > > > > >> > > > > > > > > into
> > > > > > > > >> > > > > > > > > > > > nodes
> > > > > > > > >> > > > > > > > > > > > > > > table, for addNode in Mongo,
> > > > basically
> > > > > > the
> > > > > > > > >> same
> > > > > > > > >> > > > atomic
> > > > > > > > >> > > > > > > upsert
> > > > > > > > >> > > > > > > > > > > > operation
> > > > > > > > >> > > > > > > > > > > > > > > that is currently done).
> > > Therefore,
> > > > we
> > > > > > > > >> increase
> > > > > > > > >> > > > > > performance
> > > > > > > > >> > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > Postgres
> > > > > > > > >> > > > > > > > > > > > > > > and keep the current one for
> > > Mongo.
> > > > > The
> > > > > > > > >> current
> > > > > > > > >> > DB
> > > > > > > > >> > > > > > schemas
> > > > > > > > >> > > > > > > > > won't
> > > > > > > > >> > > > > > > > > > be
> > > > > > > > >> > > > > > > > > > > > > > > touched.
> > > > > > > > >> > > > > > > > > > > > > > > Since the code change is
> large,
> > I
> > > do
> > > > > not
> > > > > > > > think
> > > > > > > > >> > I'll
> > > > > > > > >> > > > be
> > > > > > > > >> > > > > > able
> > > > > > > > >> > > > > > > > to
> > > > > > > > >> > > > > > > > > > have
> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > PR
> > > > > > > > >> > > > > > > > > > > > > > > ready till next week.
> > > > > > > > >> > > > > > > > > > > > > > > But before starting, please
> let
> > me
> > > > > know
> > > > > > if
> > > > > > > > >> that
> > > > > > > > >> > > > > approach
> > > > > > > > >> > > > > > is
> > > > > > > > >> > > > > > > > > fine
> > > > > > > > >> > > > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > you.
> > > > > > > > >> > > > > > > > > > > > > > > Best regards.
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> 6:55 PM
> > > Alex
> > > > > > > > Porcelli
> > > > > > > > >> <
> > > > > > > > >> > > > > > > > > alex@porcelli.me>
> > > > > > > > >> > > > > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > Thank you Francisco to
> getting
> > > > > deeper
> > > > > > on
> > > > > > > > >> this…
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > Looking forward to see the
> > > results
> > > > > of
> > > > > > > your
> > > > > > > > >> > > > suggested
> > > > > > > > >> > > > > > > > > > improvements.
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> > 9:40 AM
> > > > > > > Francisco
> > > > > > > > >> > Javier
> > > > > > > > >> > > > > Tirado
> > > > > > > > >> > > > > > > > > Sarti <
> > > > > > > > >> > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > I forgot to attach the
> > queries
> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> > > 3:04 PM
> > > > > > > > Francisco
> > > > > > > > >> > > Javier
> > > > > > > > >> > > > > > Tirado
> > > > > > > > >> > > > > > > > > > Sarti <
> > > > > > > > >> > > > > > > > > > > > > > > > > ftirados@redhat.com>
> wrote:
> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > > >> Hi,
> > > > > > > > >> > > > > > > > > > > > > > > > >> A brief update on this
> > topic.
> > > > > > > > >> > > > > > > > > > > > > > > > >> After doing a simple test
> > > with
> > > > > > > example
> > > > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > > >> > > > > > > > > > > > > > > > ,
> > > > > > > > >> > > > > > > > > > > > > > > > >> the number of updates
> over
> > > > Nodes
> > > > > > > table
> > > > > > > > is
> > > > > > > > >> > n*n,
> > > > > > > > >> > > > so
> > > > > > > > >> > > > > we
> > > > > > > > >> > > > > > > > > manage
> > > > > > > > >> > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > obtain a
> > > > > > > > >> > > > > > > > > > > > > > > > >> perfect quadratic
> > performance
> > > > > > > > >> degradation.
> > > > > > > > >> > The
> > > > > > > > >> > > > > > problem
> > > > > > > > >> > > > > > > > is
> > > > > > > > >> > > > > > > > > > worse
> > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > case
> > > > > > > > >> > > > > > > > > > > > > > > > >> of Serverless Workflow
> than
> > > in
> > > > > BPMN
> > > > > > > > >> because
> > > > > > > > >> > we
> > > > > > > > >> > > > the
> > > > > > > > >> > > > > > > > number
> > > > > > > > >> > > > > > > > > of
> > > > > > > > >> > > > > > > > > > > > nodes
> > > > > > > > >> > > > > > > > > > > > > > is
> > > > > > > > >> > > > > > > > > > > > > > > > >> greater than the number
> of
> > > > > states.
> > > > > > In
> > > > > > > > >> that
> > > > > > > > >> > > > > example N
> > > > > > > > >> > > > > > > is
> > > > > > > > >> > > > > > > > > 16,
> > > > > > > > >> > > > > > > > > > but
> > > > > > > > >> > > > > > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > > > a
> > > > > > > > >> > > > > > > > > > > > > > > > more
> > > > > > > > >> > > > > > > > > > > > > > > > >> complex workflow it would
> > be
> > > > > > > certainly
> > > > > > > > >> > large.
> > > > > > > > >> > > > > > > > > > > > > > > > >> I think that this is more
> > > > related
> > > > > > to
> > > > > > > > how
> > > > > > > > >> we
> > > > > > > > >> > > are
> > > > > > > > >> > > > > > > handling
> > > > > > > > >> > > > > > > > > > JPA in
> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > > > code,
> > > > > > > > >> > > > > > > > > > > > > > > > >> in particular the mapping
> > > from
> > > > > > model
> > > > > > > to
> > > > > > > > >> > entity
> > > > > > > > >> > > > > > > > (basically
> > > > > > > > >> > > > > > > > > > JPA is
> > > > > > > > >> > > > > > > > > > > > > > blind
> > > > > > > > >> > > > > > > > > > > > > > > > and
> > > > > > > > >> > > > > > > > > > > > > > > > >> has to update all nodes
> for
> > > > every
> > > > > > > write
> > > > > > > > >> > > because
> > > > > > > > >> > > > it
> > > > > > > > >> > > > > > > > > believes
> > > > > > > > >> > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > node has
> > > > > > > > >> > > > > > > > > > > > > > > > >> been updated, although it
> > is
> > > > not)
> > > > > > > than
> > > > > > > > an
> > > > > > > > >> > > issue
> > > > > > > > >> > > > in
> > > > > > > > >> > > > > > the
> > > > > > > > >> > > > > > > > > table
> > > > > > > > >> > > > > > > > > > > > > > definition.
> > > > > > > > >> > > > > > > > > > > > > > > > >> In fact, when using JPA,
> > > > > separating
> > > > > > > the
> > > > > > > > >> > server
> > > > > > > > >> > > > > model
> > > > > > > > >> > > > > > > > from
> > > > > > > > >> > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > JPA
> > > > > > > > >> > > > > > > > > > > > > > > > entity is
> > > > > > > > >> > > > > > > > > > > > > > > > >> not a good idea,
> especially
> > > if
> > > > > the
> > > > > > > > entity
> > > > > > > > >> > > > contains
> > > > > > > > >> > > > > > > > > > collections.
> > > > > > > > >> > > > > > > > > > > > I
> > > > > > > > >> > > > > > > > > > > > > > will
> > > > > > > > >> > > > > > > > > > > > > > > > try
> > > > > > > > >> > > > > > > > > > > > > > > > >> to change that without
> > > breaking
> > > > > > > > anything.
> > > > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > > > > > > > > >> On Wed, Nov 22, 2023 at
> > > > 12:10 PM
> > > > > > > > Enrique
> > > > > > > > >> > > > Gonzalez
> > > > > > > > >> > > > > > > > > Martinez <
> > > > > > > > >> > > > > > > > > > > > > > > > >> egonzalez@apache.org>
> > wrote:
> > > > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > > > > > > > > >>> After the events split
> you
> > > now
> > > > > > will
> > > > > > > > >> need to
> > > > > > > > >> > > > > create
> > > > > > > > >> > > > > > a
> > > > > > > > >> > > > > > > > node
> > > > > > > > >> > > > > > > > > > > > instance
> > > > > > > > >> > > > > > > > > > > > > > > > >>> model instance of making
> > > > > > independent
> > > > > > > > >> from
> > > > > > > > >> > the
> > > > > > > > >> > > > > > process
> > > > > > > > >> > > > > > > > > > instance.
> > > > > > > > >> > > > > > > > > > > > > > > > >>> That should do the
> trick.
> > > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > > >> > > > > > > > > > > > > > > > >>> Regarding
> > deleting/inserting
> > > > it
> > > > > > was
> > > > > > > > >> fixed
> > > > > > > > >> > at
> > > > > > > > >> > > > some
> > > > > > > > >> > > > > > > > point.
> > > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > > >> > > > > > > > > > > > > > > > >>> El mar, 21 nov 2023 a
> las
> > > > 20:22,
> > > > > > > > >> Francisco
> > > > > > > > >> > > > Javier
> > > > > > > > >> > > > > > > > Tirado
> > > > > > > > >> > > > > > > > > > Sarti
> > > > > > > > >> > > > > > > > > > > > > > > > >>> (<ft...@redhat.com>)
> > > > > escribió:
> > > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > Hi Martin,
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > I have a task to
> review
> > > > > > > performance
> > > > > > > > of
> > > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > ProcessInstanceNodeDataEventMerger
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > My idea is to reduce
> the
> > > > > number
> > > > > > of
> > > > > > > > >> delete
> > > > > > > > >> > > > > inserts
> > > > > > > > >> > > > > > > > when
> > > > > > > > >> > > > > > > > > > > > processing
> > > > > > > > >> > > > > > > > > > > > > > > > >>> events
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > and try to do it
> > > > incremental.
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > That should improve
> > > > > performance.
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > PS:
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > I was planning to send
> > an
> > > > > e-mail
> > > > > > > > >> tomorrow
> > > > > > > > >> > > > > > > announcing
> > > > > > > > >> > > > > > > > > > that in
> > > > > > > > >> > > > > > > > > > > > > > case you
> > > > > > > > >> > > > > > > > > > > > > > > > >>> were
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > already working on a
> fix
> > > for
> > > > > > > that. I
> > > > > > > > >> > assume
> > > > > > > > >> > > > you
> > > > > > > > >> > > > > > are
> > > > > > > > >> > > > > > > > not
> > > > > > > > >> > > > > > > > > > and I
> > > > > > > > >> > > > > > > > > > > > > > would
> > > > > > > > >> > > > > > > > > > > > > > > > be
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > sending a PR soon.
> > > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > > >> > > > > > > > > > > > > > > > >>> > On Tue, Nov 21, 2023
> at
> > > > > 6:09 PM
> > > > > > > > Martin
> > > > > > > > >> > > Weiler
> > > > > > > > >> > > > > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > >> > > > > > > > > > > > > > > > >>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Enrique Gonzalez Martinez <eg...@apache.org>]

Regarding your STP concerns, they were addressed in my previous comment:

STP is a concept, a process with certain constraints: no persistence and
returning the outcome in the call (sync execution with no idle states). It
was a requirement from a user in the past. One of the requirements was
leaving no trail. In v7 was easy because you could disable the audit in
that case. Actually we have the same way to do what we did in v7 in here as
you can add/remove index just removing dependencies.

So in fact in microservice you just need to exclude the data index from the
app and you wont have data index.



El mié, 21 feb 2024, 1:40, Richard Bourner <ri...@gmail.com> escribió:

> +1 with Martin's email.
>
> One question though in regards to Martin's point #3 and to previous
> following statement from Francisco: "*... keeping finishing
> process instances "for a while" in DataIndex was the only way for users to
> query the result of straight through processes*"
> --> Is this the only use case where data index would be needed for STP?
> I am asking because clients will already get their result in the JSON
> returned from the synchronous REST call, so adding an extra computing time
> for data index persistence does not seem right to me in the context of
> decision services that are supposed to be very fast to return (typically
> rule tasks+scripts).
> Or is it that we also want to provide some GraphQL capabilities, even for
> STP use cases?
>
> Also, what would "*for a while*" mean exactly?  Will it be configurable?
> Will there be a default expiration value?
>
> I am assuming this is all work in progress, and you may not have answers to
> all my questions, no problem with that.
>
> Thanks.
>
> On Tue, Feb 20, 2024 at 5:39 PM Martin Weiler <mw...@ibm.com.invalid>
> wrote:
>
> > IMO, it is good to have this discussion around data sanity now instead of
> > putting it off until later when data has already accumulated in
> production
> > environments.
> >
> > Based on the input here, we are dealing with three types of data:
> > 1. Runtime data - active instances only, engine cleans up the data
> > automatically at process instance end
> > 2. Historic log data - data created by data-audit intended for long term
> > storage
> > 3. Data-index data - somehow this data falls in between the two
> > aforementioned categories, with the idea of the data being "recent", but
> > not restricted to active instances only
> >
> > We'd need purge strategies for both #2 and #3 (perhaps different ones, or
> > with different config settings) in order to prevent unlimited data
> growth.
> >
> > ________________________________________
> > From: Enrique Gonzalez Martinez <eg...@apache.org>
> > Sent: Monday, February 19, 2024 7:11 AM
> > To: dev@kie.apache.org
> > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with data-index
> > persistence addon
> >
> > Hi Francisco,
> > To give you more context about this.
> >
> > STP is a concept, a process with certain constraints: no persistence and
> > returning the outcome in the call (sync execution with no idle states).
> It
> > was a requirement from a user in the past. One of the requirements was
> > leaving no trail. In v7 was easy because you could disable the audit in
> > that case. Actually we have the same way to do what we did in v7 in here
> as
> > you can add/remove index just removing deps.
> >
> > We have the same outcome with different approaches and STP is already
> > delivered.
> >
> > El lun, 19 feb 2024 a las 14:46, Francisco Javier Tirado Sarti (<
> > ftirados@redhat.com>) escribió:
> >
> > > Regarding STP (which is not a concept that we have in the code. I mean
> > STP
> > > are processes as nonSTP are), I guess, as all processes, they were kept
> > in
> > > DataIndex once completed because users wanted (and still wants) to
> check
> > > the result once the call had been performed. If we want to leave no
> trace
> > > of them in DataIndex for some reason, we will need to make it a
> > > Runtimes concept so DataIndex can handle them in a different way.
> > >
> > > On Mon, Feb 19, 2024 at 2:27 PM Enrique Gonzalez Martinez <
> > > egonzalez@apache.org> wrote:
> > >
> > > > Alex:
> > > > Right now the data index is working in the same way as it did in v7
> > with
> > > > the emitters. The only difference between two impl is that in here
> the
> > > > storage is pgsql instead elastic search.  You are right regarding is
> a
> > > > snapshot of the last state of the process but we did never define how
> > > long
> > > > would be alive that dats Honestly i am happy right now with the way
> it
> > > > works. The clean up mechanism is still tbd because we still need to
> > > discuss
> > > > other stuff first.
> > > >
> > > >
> > > > Regarding stp is to leave no trail because u can get the outcome
> > directly
> > > > from the call. It was defined like that in v7. So there is no use for
> > the
> > > > index or the audit.
> > > >
> > > > El lun, 19 feb 2024, 14:13, Francisco Javier Tirado Sarti <
> > > > ftirados@redhat.com> escribió:
> > > >
> > > > > Hi Alex,
> > > > > There has been some confusion about the purpose of DataIndex. To be
> > > > honest
> > > > > I believe they were already sorted out, but your e-mail makes me
> > think
> > > > that
> > > > > is not the case ;). I let Kris to clarify that with you. My view is
> > > that
> > > > > data-index is a way to query recently closed and active processes
> > (the
> > > > key
> > > > > here is the definition of recently, which in my opinion should be
> > > > > configurable)
> > > > > But, besides that discussion and being pragmatic, keeping finishing
> > > > process
> > > > > instances "for a while" in DataIndex was the only way for users to
> > > query
> > > > > the result of straight through processes. That's a function that
> > cannot
> > > > be
> > > > > removed right now
> > > > >
> > > > > On Mon, Feb 19, 2024 at 1:33 PM Alex Porcelli <porcelli@apache.org
> >
> > > > wrote:
> > > > >
> > > > > > if data index was supposed to provide snapshot view of the
> process
> > > > > > instance… why do we keep it after the process instance is
> finished?
> > > > > >
> > > > > >
> > > > > > On Mon, Feb 19, 2024 at 7:12 AM Francisco Javier Tirado Sarti <
> > > > > > ftirados@redhat.com> wrote:
> > > > > >
> > > > > > > Hi Martin.
> > > > > > > After taking a deeper look at this, I realize that the
> behaviour
> > is
> > > > the
> > > > > > > expected one.
> > > > > > > Runtimes DB does not track the completed process instance
> (that's
> > > > what
> > > > > > the
> > > > > > > JDBCProcessInstances warn is telling us), but DataIndex, as
> > > expected,
> > > > > is
> > > > > > > tracking it in processes and nodes table. And yes it will grow
> > over
> > > > > time.
> > > > > > > What we need is some configurable purge mechanism for
> DataIndex,
> > so
> > > > it
> > > > > > > eventually removes older completed process instances.
> > > > > > >
> > > > > > > On Tue, Feb 13, 2024 at 12:59 PM Francisco Javier Tirado Sarti
> <
> > > > > > > ftirados@redhat.com> wrote:
> > > > > > >
> > > > > > > > Hi Martin,
> > > > > > > > Good catch!. Looks like the skipping performed for process
> > > > instances
> > > > > is
> > > > > > > > not applied to node instances. Something we definitely need
> to
> > > > review
> > > > > > on
> > > > > > > > the runtimes side.
> > > > > > > >
> > > > > > > > On Mon, Feb 12, 2024 at 11:59 PM Martin Weiler
> > > > > <mweiler@ibm.com.invalid
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> On a somewhat related note, testing a simple workflow (start
> > ->
> > > > > script
> > > > > > > >> node -> end), I see the following messages in the logs:
> > > > > > > >> 2024-02-12 22:49:50,493 28758dde544c WARN
> > > > > > > >> [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> > > > > > > >> (executor-thread-3) Skipping create of process instance id:
> > > > > > > >> 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2
> > > > > > > >>
> > > > > > > >> So far, so good. And I'd expect to see no trace of this
> > process
> > > in
> > > > > the
> > > > > > > >> database if I don't have data audit enabled.
> > > > > > > >>
> > > > > > > >> However, the 'processes' table contains a row with state=2,
> > with
> > > > > > related
> > > > > > > >> entries in the 'nodes' table. In a load test, I see these
> > tables
> > > > > grow
> > > > > > > >> significantly over time. Am I missing something to have
> these
> > > > > entries
> > > > > > > >> cleaned up automatically?
> > > > > > > >>
> > > > > > > >> ________________________________________
> > > > > > > >> From: Martin Weiler <mw...@ibm.com.INVALID>
> > > > > > > >> Sent: Monday, February 12, 2024 3:40 PM
> > > > > > > >> To: dev@kie.apache.org
> > > > > > > >> Subject: [EXTERNAL] RE: [DISCUSSION] Performance issues with
> > > > > > data-index
> > > > > > > >> persistence addon
> > > > > > > >>
> > > > > > > >> Thanks everyone for your input. Based on this discussion, I
> > > opened
> > > > > the
> > > > > > > >> following PR:
> > > > > > > >>
> https://github.com/apache/incubator-kie-kogito-apps/pull/1985
> > > > > > > >>
> > > > > > > >> With this change, the performance seems to be stable over
> > time:
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
> > > > > > > >>
> > > > > > > >> Martin
> > > > > > > >>
> > > > > > > >> ________________________________________
> > > > > > > >> From: Gonzalo Muñoz <gm...@apache.org>
> > > > > > > >> Sent: Friday, February 9, 2024 9:42 AM
> > > > > > > >> To: dev@kie.apache.org
> > > > > > > >> Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > > > > > data-index
> > > > > > > >> persistence addon
> > > > > > > >>
> > > > > > > >> Great work Francisco,
> > > > > > > >> Martin, take a look at this link with some related tips (in
> > case
> > > > you
> > > > > > > find
> > > > > > > >> it useful):
> > > > > > > >>
> > https://www.cybertec-postgresql.com/en/index-your-foreign-key/
> > > > > > > >>
> > > > > > > >> El vie, 9 feb 2024 a las 17:20, Francisco Javier Tirado
> Sarti
> > (<
> > > > > > > >> ftirados@redhat.com>) escribió:
> > > > > > > >>
> > > > > > > >> > For the moment being, we will keep JPA till we exhaust all
> > > > > > > >> possibilities,
> > > > > > > >> > let's call switching from jpa to jdbc our hidden plan B ;)
> > > > > > > >> > I already told Martin, but in order everyone to know, just
> > > after
> > > > > > > writing
> > > > > > > >> > the previous email, I thought "what if Postgres is not
> > > > > automatically
> > > > > > > >> > indexing foreign keys like mysql?" and, eureka
> > > > > > > >> > Postgres doc
> > > > > > > >> >
> > https://www.postgresql.org/docs/current/ddl-constraints.html
> > > > > > > >> > Mysql doc
> > > > > > > >> >
> > > > >
> https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> > > > > > > >> > These are the relevant excerpt
> > > > > > > >> >
> > > > > > > >> > *Postgresql*
> > > > > > > >> > *A foreign key must reference columns that either are a
> > > primary
> > > > > key
> > > > > > or
> > > > > > > >> form
> > > > > > > >> > a unique constraint, or are columns from a non-partial
> > unique
> > > > > index.
> > > > > > > >> This
> > > > > > > >> > means that the referenced columns always have an index to
> > > allow
> > > > > > > >> efficient
> > > > > > > >> > lookups on whether a referencing row has a match. Since a
> > > DELETE
> > > > > of
> > > > > > a
> > > > > > > >> row
> > > > > > > >> > from the referenced table or an UPDATE of a referenced
> > column
> > > > will
> > > > > > > >> require
> > > > > > > >> > a scan of the referencing table for rows matching the old
> > > value,
> > > > > it
> > > > > > is
> > > > > > > >> > often a good idea to index the referencing columns too.
> > > Because
> > > > > this
> > > > > > > is
> > > > > > > >> not
> > > > > > > >> > always needed, and there are many choices available on how
> > to
> > > > > index,
> > > > > > > the
> > > > > > > >> > declaration of a foreign key constraint does not
> > automatically
> > > > > > create
> > > > > > > an
> > > > > > > >> > index on the referencing columns.*
> > > > > > > >> > *Mysql*
> > > > > > > >> > *MySQL requires that foreign key columns be indexed; if
> you
> > > > > create a
> > > > > > > >> table
> > > > > > > >> > with a foreign key constraint but no index on a given
> > column,
> > > an
> > > > > > index
> > > > > > > >> is
> > > > > > > >> > created. *
> > > > > > > >> >
> > > > > > > >> > So I asked Martin to especially create an index for
> > > > > > > process_instance_id
> > > > > > > >> > column on nodes table
> > > > > > > >> > I think that will fix the problem detected on the thread
> > dump.
> > > > > > > >> > The simpler process test to verify queries are fine still
> > > > stands,
> > > > > > > >> though ;)
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <
> > > > tzimanyi@apache.org
> > > > > >
> > > > > > > >> wrote:
> > > > > > > >> >
> > > > > > > >> > > I always preferred pure JDBC over Hibernate myself, just
> > for
> > > > the
> > > > > > > sake
> > > > > > > >> of
> > > > > > > >> > > control of what is happening :) So I would not -1 that
> > > myself.
> > > > > > > >> > >
> > > > > > > >> > > Tibor
> > > > > > > >> > >
> > > > > > > >> > > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado Sarti <
> > > > > > > >> > > ftirados@redhat.com>
> > > > > > > >> > > napísal(a):
> > > > > > > >> > >
> > > > > > > >> > > > Hi,
> > > > > > > >> > > > Usually I do not want to talk about work in progress
> > > because
> > > > > > > >> > preliminary
> > > > > > > >> > > > conclusions are pretty volatile but, well, there are a
> > > > couple
> > > > > of
> > > > > > > >> things
> > > > > > > >> > > > that can be concluded from the really valuable
> > information
> > > > > that
> > > > > > > >> Martin
> > > > > > > >> > > > provided.
> > > > > > > >> > > > 1) In order to be able to determine if the number of
> > > > > statements
> > > > > > is
> > > > > > > >> > larger
> > > > > > > >> > > > than expected, I asked Martin to test with a simpler
> > > process
> > > > > > > >> > definition.
> > > > > > > >> > > > One with just three nodes: start, script and end. The
> > > script
> > > > > one
> > > > > > > >> should
> > > > > > > >> > > > change just one variable. This way we can analyze if
> the
> > > > > number
> > > > > > of
> > > > > > > >> > > queries
> > > > > > > >> > > > is the expected one. From the single log (audit was
> > > > activated
> > > > > > > them)
> > > > > > > >> my
> > > > > > > >> > > > conclusion is that the number of insert/updates over
> > > > processes
> > > > > > and
> > > > > > > >> > nodes
> > > > > > > >> > > > (there a lot over task, that I will prefer to skip for
> > > now,
> > > > > baby
> > > > > > > >> steps)
> > > > > > > >> > > is
> > > > > > > >> > > > the expected one.
> > > > > > > >> > > > 2) Analysing the thread dump, we see around 15 threads
> > > > > executing
> > > > > > > >> this
> > > > > > > >> > > line
> > > > > > > >> > > > at
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > > > > > >> > > > so its pretty clear the code to be optimized ;). I'm
> > > > > evaluating
> > > > > > > >> > > > possibilities within JPA/Hibernate, but I'm starting
> to
> > > > think
> > > > > > that
> > > > > > > >> it
> > > > > > > >> > > might
> > > > > > > >> > > > be better to switch to JDBC and skip hibernate. Our
> > lives
> > > > will
> > > > > > be
> > > > > > > >> > > simpler,
> > > > > > > >> > > > especially with a schema relatively simple like ours
> > (that
> > > > > will
> > > > > > be
> > > > > > > >> my
> > > > > > > >> > > > recommendation if I was an external consultant)
> > > > > > > >> > > >
> > > > > > > >> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <
> > > > > > tzimanyi@apache.org
> > > > > > > >
> > > > > > > >> > > wrote:
> > > > > > > >> > > >
> > > > > > > >> > > > > Hi,
> > > > > > > >> > > > >
> > > > > > > >> > > > > this will be a bit off-topic. However as far as
> > > > > performance, I
> > > > > > > >> think
> > > > > > > >> > we
> > > > > > > >> > > > > should think about that we have string primary keys
> > > > (IDs). I
> > > > > > > would
> > > > > > > >> > > expect
> > > > > > > >> > > > > the database systems are much better with indexing
> > > numeric
> > > > > > keys
> > > > > > > >> than
> > > > > > > >> > > > > strings. I remember from the past, when I was
> working
> > > with
> > > > > > DBs,
> > > > > > > >> that
> > > > > > > >> > > > using
> > > > > > > >> > > > > strings as keys or indexes was a discouraged
> practice.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Best regards,
> > > > > > > >> > > > > Tibor
> > > > > > > >> > > > >
> > > > > > > >> > > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler
> > > > > > <mweiler@ibm.com.invalid
> > > > > > > >
> > > > > > > >> > > > > napísal(a):
> > > > > > > >> > > > >
> > > > > > > >> > > > > > I changed the test to use MongoDB [1] and I don't
> > see
> > > a
> > > > > > > >> performance
> > > > > > > >> > > > > > degradation with this setup [2].
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Please keep us posted of your findings. Thanks!
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Martin
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > [1]
> > > > > > > >> > > > >
> > > > > > > >> >
> > > > > > >
> > > >
> https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > > > > >> > > > > > [2]
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > ________________________________________
> > > > > > > >> > > > > > From: Francisco Javier Tirado Sarti <
> > > > ftirados@redhat.com>
> > > > > > > >> > > > > > Sent: Wednesday, February 7, 2024 11:40 AM
> > > > > > > >> > > > > > To: dev@kie.apache.org
> > > > > > > >> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance
> > > issues
> > > > > with
> > > > > > > >> > > data-index
> > > > > > > >> > > > > > persistence addon
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > yes, it can be index degradation because of size,
> > but
> > > I
> > > > > > > believe
> > > > > > > >> (I
> > > > > > > >> > > > might
> > > > > > > >> > > > > be
> > > > > > > >> > > > > > wrong) the db is too small (yet) for that.
> > > > > > > >> > > > > > But, eventually, Postgres, when the DB is huge
> > enough,
> > > > > > > >> unavoidably
> > > > > > > >> > > will
> > > > > > > >> > > > > > behave like the graphic that Martin sent.
> > > > > > > >> > > > > > Since I believe we are not huge enough (yet), lets
> > > rule
> > > > > out
> > > > > > > >> another
> > > > > > > >> > > > issue
> > > > > > > >> > > > > > by analysing the sql logs (I requested those to
> > Martin
> > > > > > offline
> > > > > > > >> and
> > > > > > > >> > he
> > > > > > > >> > > > is
> > > > > > > >> > > > > > going to kindly collect them).
> > > > > > > >> > > > > > Also Im curious to know if Mongo behave in the
> same
> > > way.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez
> > > > Martinez <
> > > > > > > >> > > > > > egonzalez@apache.org> wrote:
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > Hi Francisco,
> > > > > > > >> > > > > > > I would highly recommend to check indexes and
> how
> > > the
> > > > > > > updates
> > > > > > > >> > work
> > > > > > > >> > > in
> > > > > > > >> > > > > > data
> > > > > > > >> > > > > > > index to avoid full scan table and lock the full
> > > > table.
> > > > > > Some
> > > > > > > >> db
> > > > > > > >> > are
> > > > > > > >> > > > > very
> > > > > > > >> > > > > > > sensitive to that.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > El mié, 7 feb 2024, 18:41, Francisco Javier
> Tirado
> > > > > Sarti <
> > > > > > > >> > > > > > > ftirados@redhat.com> escribió:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > > Hi Martin,
> > > > > > > >> > > > > > > > While I analyze the data, let me ask you if it
> > is
> > > > > > possible
> > > > > > > >> to
> > > > > > > >> > > > perform
> > > > > > > >> > > > > > > > another check (similar in a way to disabling
> > > > > data-index
> > > > > > > like
> > > > > > > >> > you
> > > > > > > >> > > > do)
> > > > > > > >> > > > > > Can
> > > > > > > >> > > > > > > > you switch to MongoDB persistence and check if
> > the
> > > > > same
> > > > > > > >> > > degradation
> > > > > > > >> > > > > > that
> > > > > > > >> > > > > > > is
> > > > > > > >> > > > > > > > there for postgres remains?
> > > > > > > >> > > > > > > > I do not know if this is feasible but will
> > > certainly
> > > > > > > >> indicate
> > > > > > > >> > the
> > > > > > > >> > > > > > problem
> > > > > > > >> > > > > > > > is on the postgres storage layer and I do not
> > > have a
> > > > > > clear
> > > > > > > >> > > > prediction
> > > > > > > >> > > > > > of
> > > > > > > >> > > > > > > > what we will see when doing this switch.
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler
> > > > > > > >> > > > <mweiler@ibm.com.invalid
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > > wrote:
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > > Hi Francisco,
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > thanks for your work on this important
> topic!
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > I would like to share some test results
> here,
> > > > which
> > > > > > > might
> > > > > > > >> > help
> > > > > > > >> > > to
> > > > > > > >> > > > > > > improve
> > > > > > > >> > > > > > > > > the codebase even further. I am using the
> > jmeter
> > > > > based
> > > > > > > >> test
> > > > > > > >> > > case
> > > > > > > >> > > > > from
> > > > > > > >> > > > > > > > Pere
> > > > > > > >> > > > > > > > > and Enrique (thanks guys!) [1] which uses a
> > load
> > > > of
> > > > > 30
> > > > > > > >> > threads
> > > > > > > >> > > to
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > 1) start a new process instance (POST)
> > > > > > > >> > > > > > > > > 2) retrieve tasks for a user (GET)
> > > > > > > >> > > > > > > > > 3) fetches task details (GET)
> > > > > > > >> > > > > > > > > 4) complete a task (POST)
> > > > > > > >> > > > > > > > > 5) execute a query on data-audit
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > With this test setup, I noticed that the
> > > > performance
> > > > > > for
> > > > > > > >> the
> > > > > > > >> > > POST
> > > > > > > >> > > > > > > > > requests, in particular the one to start a
> new
> > > > > process
> > > > > > > >> > > instance,
> > > > > > > >> > > > > > > degrades
> > > > > > > >> > > > > > > > > over time - see graph [2]. If I run the same
> > > test
> > > > > > > without
> > > > > > > >> > > > > data-index,
> > > > > > > >> > > > > > > > then
> > > > > > > >> > > > > > > > > there is no such performance degradation
> [3].
> > > You
> > > > > can
> > > > > > > >> find a
> > > > > > > >> > > > thread
> > > > > > > >> > > > > > > dump
> > > > > > > >> > > > > > > > > captured a few minutes into the first test
> > here
> > > > [4]
> > > > > > that
> > > > > > > >> > might
> > > > > > > >> > > > help
> > > > > > > >> > > > > > to
> > > > > > > >> > > > > > > > see
> > > > > > > >> > > > > > > > > some of the contention points.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > I'd appreciate if you could take a look and
> > see
> > > if
> > > > > > there
> > > > > > > >> is
> > > > > > > >> > > > > something
> > > > > > > >> > > > > > > > that
> > > > > > > >> > > > > > > > > can be further improved based on your
> previous
> > > > work.
> > > > > > If
> > > > > > > >> you
> > > > > > > >> > > need
> > > > > > > >> > > > > any
> > > > > > > >> > > > > > > > > additional data, let me know, but otherwise
> it
> > > is
> > > > > > > >> > > straightforward
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > > run
> > > > > > > >> > > > > > > > > the jmeter test as well.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Thanks,
> > > > > > > >> > > > > > > > > Martin
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > [1]
> > > > > > > >> https://github.com/pefernan/job-service-refactor-test/
> > > > > > > >> > > > > > > > > [2]
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > > > >> > > > > > > > > [3]
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > > > >> > > > > > > > > [4]
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > ________________________________________
> > > > > > > >> > > > > > > > > From: Francisco Javier Tirado Sarti <
> > > > > > > ftirados@redhat.com>
> > > > > > > >> > > > > > > > > Sent: Wednesday, January 17, 2024 9:13 AM
> > > > > > > >> > > > > > > > > To: dev@kie.apache.org
> > > > > > > >> > > > > > > > > Cc: Pere Fernandez Perez
> > > > > > > >> > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION]
> > Performance
> > > > > > issues
> > > > > > > >> with
> > > > > > > >> > > > > > data-index
> > > > > > > >> > > > > > > > > persistence addon
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Hi Alex,
> > > > > > > >> > > > > > > > > I did not take times (which depends on a
> > number
> > > of
> > > > > > > >> variables
> > > > > > > >> > > that
> > > > > > > >> > > > > > > > > drastically change between environments),
> but
> > > > verify
> > > > > > > that
> > > > > > > >> the
> > > > > > > >> > > > > number
> > > > > > > >> > > > > > of
> > > > > > > >> > > > > > > > > updates has been reduced drastically without
> > > > losing
> > > > > > > >> > > > functionality,
> > > > > > > >> > > > > > > which
> > > > > > > >> > > > > > > > is
> > > > > > > >> > > > > > > > > objectively a good thing. If before the
> > change,
> > > > for
> > > > > > > every
> > > > > > > >> > node
> > > > > > > >> > > > > > > executed,
> > > > > > > >> > > > > > > > we
> > > > > > > >> > > > > > > > > have an update for every node previously
> > > executed,
> > > > > so
> > > > > > > if a
> > > > > > > >> > > > process
> > > > > > > >> > > > > > have
> > > > > > > >> > > > > > > > 50
> > > > > > > >> > > > > > > > > nodes to execute, we were performing nearly
> > > > 50*51/2
> > > > > > > >> updates,
> > > > > > > >> > > > which
> > > > > > > >> > > > > > > gives
> > > > > > > >> > > > > > > > us
> > > > > > > >> > > > > > > > > a total of  1275 updates, now we have just
> one
> > > for
> > > > > > every
> > > > > > > >> node
> > > > > > > >> > > > being
> > > > > > > >> > > > > > > > > executed, implying a total of 50 updates.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex
> Porcelli
> > <
> > > > > > > >> > > alex@porcelli.me>
> > > > > > > >> > > > > > > wrote:
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > > Francisco,
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > I noticed that your PR has been merged,
> but
> > I
> > > > was
> > > > > > > >> expecting
> > > > > > > >> > > (at
> > > > > > > >> > > > > > least
> > > > > > > >> > > > > > > > > > was my understanding from this thread)
> that
> > > > before
> > > > > > > >> merging
> > > > > > > >> > > some
> > > > > > > >> > > > > > > > > > benchmark data would be shared in advance
> -
> > to
> > > > > > assess
> > > > > > > >> the
> > > > > > > >> > > > > > > cost/benefit
> > > > > > > >> > > > > > > > > > of such a decent size change.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Do you have any information to share?
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM Francisco
> > > Javier
> > > > > > > Tirado
> > > > > > > >> > Sarti
> > > > > > > >> > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Yes, as intended, now we have one select
> > and
> > > > one
> > > > > > > >> > > > insert/update
> > > > > > > >> > > > > > per
> > > > > > > >> > > > > > > > node
> > > > > > > >> > > > > > > > > > > event.
> > > > > > > >> > > > > > > > > > > I moved the PR as ready for review and
> > give
> > > > > @Pere
> > > > > > > >> > Fernandez
> > > > > > > >> > > > > Perez
> > > > > > > >> > > > > > > > > > > <pe...@redhat.com> permission to the
> > > > branch
> > > > > so
> > > > > > > he
> > > > > > > >> can
> > > > > > > >> > > > edit
> > > > > > > >> > > > > it
> > > > > > > >> > > > > > > in
> > > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > > > > > > next two weeks (Ill be on PTO)  if
> > desired,
> > > > > before
> > > > > > > >> > merging.
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex
> > > Porcelli
> > > > <
> > > > > > > >> > > > > alex@porcelli.me>
> > > > > > > >> > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > Cool, thank you Francisco!
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > Did you manage to get some preliminary
> > > data
> > > > > > about
> > > > > > > >> > > > > improvements?
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM
> > Francisco
> > > > > > Javier
> > > > > > > >> > Tirado
> > > > > > > >> > > > > Sarti
> > > > > > > >> > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > Yes, after some delay because of
> > > quarkus 3
> > > > > > > >> migration.
> > > > > > > >> > > Im
> > > > > > > >> > > > > > > refining
> > > > > > > >> > > > > > > > > > this
> > > > > > > >> > > > > > > > > > > > > draft PR
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > >
> > > > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex
> > > > > Porcelli
> > > > > > <
> > > > > > > >> > > > > > > alex@porcelli.me>
> > > > > > > >> > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > Any update or new findings on this
> > > > topic?
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM
> > > > Francisco
> > > > > > > Javier
> > > > > > > >> > > Tirado
> > > > > > > >> > > > > > Sarti
> > > > > > > >> > > > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > Hi Alex,
> > > > > > > >> > > > > > > > > > > > > > > After considering different
> > options
> > > to
> > > > > > > improve
> > > > > > > >> > > > > > performance,
> > > > > > > >> > > > > > > > we
> > > > > > > >> > > > > > > > > > feel
> > > > > > > >> > > > > > > > > > > > that
> > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > >> > > > > > > > > > > > > > > is time to "partially" move away
> > > from
> > > > > the
> > > > > > > >> current
> > > > > > > >> > > Map
> > > > > > > >> > > > > > style
> > > > > > > >> > > > > > > > > > > > interface (
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > > >> > > > > > > > > > > > > > )
> > > > > > > >> > > > > > > > > > > > > > > which was shared with Trusty, to
> > one
> > > > > more
> > > > > > > >> > suitable
> > > > > > > >> > > > for
> > > > > > > >> > > > > > > usage
> > > > > > > >> > > > > > > > > > with a
> > > > > > > >> > > > > > > > > > > > > > > relational DB like postgresql
> (but
> > > > still
> > > > > > > >> > compatible
> > > > > > > >> > > > > with
> > > > > > > >> > > > > > > big
> > > > > > > >> > > > > > > > > > table
> > > > > > > >> > > > > > > > > > > > dbs).
> > > > > > > >> > > > > > > > > > > > > > > The idea will be to replace
> > generic
> > > > > > Storage
> > > > > > > >> > > interface
> > > > > > > >> > > > > by
> > > > > > > >> > > > > > > four
> > > > > > > >> > > > > > > > > > > > specific
> > > > > > > >> > > > > > > > > > > > > > > interfaces (which will inherit
> > from
> > > a
> > > > > > common
> > > > > > > >> one
> > > > > > > >> > > that
> > > > > > > >> > > > > > keeps
> > > > > > > >> > > > > > > > the
> > > > > > > >> > > > > > > > > > query
> > > > > > > >> > > > > > > > > > > > > > part
> > > > > > > >> > > > > > > > > > > > > > > at is it. with get and query
> > > methods),
> > > > > > that
> > > > > > > >> will
> > > > > > > >> > > > > include
> > > > > > > >> > > > > > > the
> > > > > > > >> > > > > > > > > > required
> > > > > > > >> > > > > > > > > > > > > > > modification operations for the
> > four
> > > > > > > DataIndex
> > > > > > > >> > > > > "domains":
> > > > > > > >> > > > > > > > > > > > > > processinstance,
> > > > > > > >> > > > > > > > > > > > > > > usertask, processdefinitions and
> > > jobs.
> > > > > > Those
> > > > > > > >> > > > interfaces
> > > > > > > >> > > > > > > will
> > > > > > > >> > > > > > > > > > define
> > > > > > > >> > > > > > > > > > > > > > methods
> > > > > > > >> > > > > > > > > > > > > > > like addNode, addVariable,
> > > updateTask,
> > > > > > > >> > > > > addAttachment.....
> > > > > > > >> > > > > > > > that
> > > > > > > >> > > > > > > > > > will
> > > > > > > >> > > > > > > > > > > > allow
> > > > > > > >> > > > > > > > > > > > > > > the persistent layer
> > implementation
> > > > to
> > > > > > just
> > > > > > > >> > update
> > > > > > > >> > > > the
> > > > > > > >> > > > > > > > needed
> > > > > > > >> > > > > > > > > > info
> > > > > > > >> > > > > > > > > > > > in
> > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > DB  (for example, for addNode in
> > > > > Postgres,
> > > > > > > >> just
> > > > > > > >> > > > insert
> > > > > > > >> > > > > a
> > > > > > > >> > > > > > > row
> > > > > > > >> > > > > > > > > into
> > > > > > > >> > > > > > > > > > > > nodes
> > > > > > > >> > > > > > > > > > > > > > > table, for addNode in Mongo,
> > > basically
> > > > > the
> > > > > > > >> same
> > > > > > > >> > > > atomic
> > > > > > > >> > > > > > > upsert
> > > > > > > >> > > > > > > > > > > > operation
> > > > > > > >> > > > > > > > > > > > > > > that is currently done).
> > Therefore,
> > > we
> > > > > > > >> increase
> > > > > > > >> > > > > > performance
> > > > > > > >> > > > > > > > for
> > > > > > > >> > > > > > > > > > > > Postgres
> > > > > > > >> > > > > > > > > > > > > > > and keep the current one for
> > Mongo.
> > > > The
> > > > > > > >> current
> > > > > > > >> > DB
> > > > > > > >> > > > > > schemas
> > > > > > > >> > > > > > > > > won't
> > > > > > > >> > > > > > > > > > be
> > > > > > > >> > > > > > > > > > > > > > > touched.
> > > > > > > >> > > > > > > > > > > > > > > Since the code change is large,
> I
> > do
> > > > not
> > > > > > > think
> > > > > > > >> > I'll
> > > > > > > >> > > > be
> > > > > > > >> > > > > > able
> > > > > > > >> > > > > > > > to
> > > > > > > >> > > > > > > > > > have
> > > > > > > >> > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > PR
> > > > > > > >> > > > > > > > > > > > > > > ready till next week.
> > > > > > > >> > > > > > > > > > > > > > > But before starting, please let
> me
> > > > know
> > > > > if
> > > > > > > >> that
> > > > > > > >> > > > > approach
> > > > > > > >> > > > > > is
> > > > > > > >> > > > > > > > > fine
> > > > > > > >> > > > > > > > > > for
> > > > > > > >> > > > > > > > > > > > you.
> > > > > > > >> > > > > > > > > > > > > > > Best regards.
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 6:55 PM
> > Alex
> > > > > > > Porcelli
> > > > > > > >> <
> > > > > > > >> > > > > > > > > alex@porcelli.me>
> > > > > > > >> > > > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > Thank you Francisco to getting
> > > > deeper
> > > > > on
> > > > > > > >> this…
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > Looking forward to see the
> > results
> > > > of
> > > > > > your
> > > > > > > >> > > > suggested
> > > > > > > >> > > > > > > > > > improvements.
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> 9:40 AM
> > > > > > Francisco
> > > > > > > >> > Javier
> > > > > > > >> > > > > Tirado
> > > > > > > >> > > > > > > > > Sarti <
> > > > > > > >> > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > I forgot to attach the
> queries
> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> > 3:04 PM
> > > > > > > Francisco
> > > > > > > >> > > Javier
> > > > > > > >> > > > > > Tirado
> > > > > > > >> > > > > > > > > > Sarti <
> > > > > > > >> > > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > > >> Hi,
> > > > > > > >> > > > > > > > > > > > > > > > >> A brief update on this
> topic.
> > > > > > > >> > > > > > > > > > > > > > > > >> After doing a simple test
> > with
> > > > > > example
> > > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > >> > > > > > > > > > > > > > > > ,
> > > > > > > >> > > > > > > > > > > > > > > > >> the number of updates over
> > > Nodes
> > > > > > table
> > > > > > > is
> > > > > > > >> > n*n,
> > > > > > > >> > > > so
> > > > > > > >> > > > > we
> > > > > > > >> > > > > > > > > manage
> > > > > > > >> > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > obtain a
> > > > > > > >> > > > > > > > > > > > > > > > >> perfect quadratic
> performance
> > > > > > > >> degradation.
> > > > > > > >> > The
> > > > > > > >> > > > > > problem
> > > > > > > >> > > > > > > > is
> > > > > > > >> > > > > > > > > > worse
> > > > > > > >> > > > > > > > > > > > in
> > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > case
> > > > > > > >> > > > > > > > > > > > > > > > >> of Serverless Workflow than
> > in
> > > > BPMN
> > > > > > > >> because
> > > > > > > >> > we
> > > > > > > >> > > > the
> > > > > > > >> > > > > > > > number
> > > > > > > >> > > > > > > > > of
> > > > > > > >> > > > > > > > > > > > nodes
> > > > > > > >> > > > > > > > > > > > > > is
> > > > > > > >> > > > > > > > > > > > > > > > >> greater than the number of
> > > > states.
> > > > > In
> > > > > > > >> that
> > > > > > > >> > > > > example N
> > > > > > > >> > > > > > > is
> > > > > > > >> > > > > > > > > 16,
> > > > > > > >> > > > > > > > > > but
> > > > > > > >> > > > > > > > > > > > for
> > > > > > > >> > > > > > > > > > > > > > a
> > > > > > > >> > > > > > > > > > > > > > > > more
> > > > > > > >> > > > > > > > > > > > > > > > >> complex workflow it would
> be
> > > > > > certainly
> > > > > > > >> > large.
> > > > > > > >> > > > > > > > > > > > > > > > >> I think that this is more
> > > related
> > > > > to
> > > > > > > how
> > > > > > > >> we
> > > > > > > >> > > are
> > > > > > > >> > > > > > > handling
> > > > > > > >> > > > > > > > > > JPA in
> > > > > > > >> > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > > > code,
> > > > > > > >> > > > > > > > > > > > > > > > >> in particular the mapping
> > from
> > > > > model
> > > > > > to
> > > > > > > >> > entity
> > > > > > > >> > > > > > > > (basically
> > > > > > > >> > > > > > > > > > JPA is
> > > > > > > >> > > > > > > > > > > > > > blind
> > > > > > > >> > > > > > > > > > > > > > > > and
> > > > > > > >> > > > > > > > > > > > > > > > >> has to update all nodes for
> > > every
> > > > > > write
> > > > > > > >> > > because
> > > > > > > >> > > > it
> > > > > > > >> > > > > > > > > believes
> > > > > > > >> > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > node has
> > > > > > > >> > > > > > > > > > > > > > > > >> been updated, although it
> is
> > > not)
> > > > > > than
> > > > > > > an
> > > > > > > >> > > issue
> > > > > > > >> > > > in
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > > > > table
> > > > > > > >> > > > > > > > > > > > > > definition.
> > > > > > > >> > > > > > > > > > > > > > > > >> In fact, when using JPA,
> > > > separating
> > > > > > the
> > > > > > > >> > server
> > > > > > > >> > > > > model
> > > > > > > >> > > > > > > > from
> > > > > > > >> > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > JPA
> > > > > > > >> > > > > > > > > > > > > > > > entity is
> > > > > > > >> > > > > > > > > > > > > > > > >> not a good idea, especially
> > if
> > > > the
> > > > > > > entity
> > > > > > > >> > > > contains
> > > > > > > >> > > > > > > > > > collections.
> > > > > > > >> > > > > > > > > > > > I
> > > > > > > >> > > > > > > > > > > > > > will
> > > > > > > >> > > > > > > > > > > > > > > > try
> > > > > > > >> > > > > > > > > > > > > > > > >> to change that without
> > breaking
> > > > > > > anything.
> > > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > > >> > > > > > > > > > > > > > > > >> On Wed, Nov 22, 2023 at
> > > 12:10 PM
> > > > > > > Enrique
> > > > > > > >> > > > Gonzalez
> > > > > > > >> > > > > > > > > Martinez <
> > > > > > > >> > > > > > > > > > > > > > > > >> egonzalez@apache.org>
> wrote:
> > > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > > >> > > > > > > > > > > > > > > > >>> After the events split you
> > now
> > > > > will
> > > > > > > >> need to
> > > > > > > >> > > > > create
> > > > > > > >> > > > > > a
> > > > > > > >> > > > > > > > node
> > > > > > > >> > > > > > > > > > > > instance
> > > > > > > >> > > > > > > > > > > > > > > > >>> model instance of making
> > > > > independent
> > > > > > > >> from
> > > > > > > >> > the
> > > > > > > >> > > > > > process
> > > > > > > >> > > > > > > > > > instance.
> > > > > > > >> > > > > > > > > > > > > > > > >>> That should do the trick.
> > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > >> > > > > > > > > > > > > > > > >>> Regarding
> deleting/inserting
> > > it
> > > > > was
> > > > > > > >> fixed
> > > > > > > >> > at
> > > > > > > >> > > > some
> > > > > > > >> > > > > > > > point.
> > > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > > >> > > > > > > > > > > > > > > > >>> El mar, 21 nov 2023 a las
> > > 20:22,
> > > > > > > >> Francisco
> > > > > > > >> > > > Javier
> > > > > > > >> > > > > > > > Tirado
> > > > > > > >> > > > > > > > > > Sarti
> > > > > > > >> > > > > > > > > > > > > > > > >>> (<ft...@redhat.com>)
> > > > escribió:
> > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > >> > > > > > > > > > > > > > > > >>> > Hi Martin,
> > > > > > > >> > > > > > > > > > > > > > > > >>> > I have a task to review
> > > > > > performance
> > > > > > > of
> > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > ProcessInstanceNodeDataEventMerger
> > > > > > > >> > > > > > > > > > > > > > > > >>> > My idea is to reduce the
> > > > number
> > > > > of
> > > > > > > >> delete
> > > > > > > >> > > > > inserts
> > > > > > > >> > > > > > > > when
> > > > > > > >> > > > > > > > > > > > processing
> > > > > > > >> > > > > > > > > > > > > > > > >>> events
> > > > > > > >> > > > > > > > > > > > > > > > >>> > and try to do it
> > > incremental.
> > > > > > > >> > > > > > > > > > > > > > > > >>> > That should improve
> > > > performance.
> > > > > > > >> > > > > > > > > > > > > > > > >>> > PS:
> > > > > > > >> > > > > > > > > > > > > > > > >>> > I was planning to send
> an
> > > > e-mail
> > > > > > > >> tomorrow
> > > > > > > >> > > > > > > announcing
> > > > > > > >> > > > > > > > > > that in
> > > > > > > >> > > > > > > > > > > > > > case you
> > > > > > > >> > > > > > > > > > > > > > > > >>> were
> > > > > > > >> > > > > > > > > > > > > > > > >>> > already working on a fix
> > for
> > > > > > that. I
> > > > > > > >> > assume
> > > > > > > >> > > > you
> > > > > > > >> > > > > > are
> > > > > > > >> > > > > > > > not
> > > > > > > >> > > > > > > > > > and I
> > > > > > > >> > > > > > > > > > > > > > would
> > > > > > > >> > > > > > > > > > > > > > > > be
> > > > > > > >> > > > > > > > > > > > > > > > >>> > sending a PR soon.
> > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > >> > > > > > > > > > > > > > > > >>> > On Tue, Nov 21, 2023 at
> > > > 6:09 PM
> > > > > > > Martin
> > > > > > > >> > > Weiler
> > > > > > > >> > > > > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > >> > > > > > > > > > > > > > > > >>> > wrote:
> > > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > I looked into the new
> > > > examples
> > > > > > > using
> > > > > > > >> > > > > data-index
> > > > > > > >> > > > > > > > > > persistence
> > > > > > > >> > > > > > > > > > > > > > addon -
> > > > > > > >> > > > > > > > > > > > > > > > >>> Neus'
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > PR#1813 [1] for
> > serverless
> > > > and
> > > > > > > >> Pere's
> > > > > > > >> > > > branch
> > > > > > > >> > > > > > [2]
> > > > > > > >> > > > > > > > for
> > > > > > > >> > > > > > > > > > > > workflow
> > > > > > > >> > > > > > > > > > > > > > > > (great
> > > > > > > >> > > > > > > > > > > > > > > > >>> job
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > both!) - and they work
> > > > without
> > > > > > > >> issues
> > > > > > > >> > > using
> > > > > > > >> > > > > > > single
> > > > > > > >> > > > > > > > > > > > requests.
> > > > > > > >> > > > > > > > > > > > > > > > >>> However, under
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > some load (I used 'ab'
> > for
> > > > > > testing
> > > > > > > >> > with a
> > > > > > > >> > > > > light
> > > > > > > >> > > > > > > > > > > > concurrency of
> > > > > > > >> > > > > > > > > > > > > > 10
> > > > > > > >> > > > > > > > > > > > > > > > >>> parallel
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > requests) I ran into
> the
> > > > > > following
> > > > > > > >> > > > problems:
> > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > (1) Large number of
> > > > > > insert/delete
> > > > > > > >> calls
> > > > > > > >> > > > (eg.
> > > > > > > >> > > > > > for
> > > > > > > >> > > > > > > > > tables
> > > > > > > >> > > > > > > > > > > > such as
> > > > > > > >> > > > > > > > > > > > > > > > >>> nodes,
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > definitions, etc)
> > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > (2) Hibernate
> > > > > > > >> OptimisticLockExceptions
> > > > > > > >> > /
> > > > > > > >> > > > > > > > > > > > StaleStateExceptions
> > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > (3) DB deadlocks
> > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > (4) Error responses,
> > slow
> > > > > > response
> > > > > > > >> > times
> > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > The reason I am
> reaching
> > > out
> > > > > > with
> > > > > > > >> this
> > > > > > > >> > > > topic
> > > > > > > >> > > > > > here
> > > > > > > >> > > > > > > > is
> > > > > > > >> > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > find
> > > > > > > >> > > > > > > > > > > > > > out if
> > > > > > > >> > > > > > > > > > > > > > > > >>> we are
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > aware of this issue,
> and
> > > if
> > > > > > > someone
> > > > > > > >> is
> > > > > > > >> > > > > already
> > > > > > > >> > > > > > > > > looking
> > > > > > > >> > > > > > > > > > > > into or
> > > > > > > >> > > > > > > > > > > > > > > > being
> > > > > > > >> > > > > > > > > > > > > > > > >>> > > assigned to it?
> > > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > > >> > > > > > > > >

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Richard Bourner <ri...@gmail.com>]

+1 with Martin's email.

One question though in regards to Martin's point #3 and to previous
following statement from Francisco: "*... keeping finishing
process instances "for a while" in DataIndex was the only way for users to
query the result of straight through processes*"
--> Is this the only use case where data index would be needed for STP?
I am asking because clients will already get their result in the JSON
returned from the synchronous REST call, so adding an extra computing time
for data index persistence does not seem right to me in the context of
decision services that are supposed to be very fast to return (typically
rule tasks+scripts).
Or is it that we also want to provide some GraphQL capabilities, even for
STP use cases?

Also, what would "*for a while*" mean exactly?  Will it be configurable?
Will there be a default expiration value?

I am assuming this is all work in progress, and you may not have answers to
all my questions, no problem with that.

Thanks.

On Tue, Feb 20, 2024 at 5:39 PM Martin Weiler <mw...@ibm.com.invalid>
wrote:

> IMO, it is good to have this discussion around data sanity now instead of
> putting it off until later when data has already accumulated in production
> environments.
>
> Based on the input here, we are dealing with three types of data:
> 1. Runtime data - active instances only, engine cleans up the data
> automatically at process instance end
> 2. Historic log data - data created by data-audit intended for long term
> storage
> 3. Data-index data - somehow this data falls in between the two
> aforementioned categories, with the idea of the data being "recent", but
> not restricted to active instances only
>
> We'd need purge strategies for both #2 and #3 (perhaps different ones, or
> with different config settings) in order to prevent unlimited data growth.
>
> ________________________________________
> From: Enrique Gonzalez Martinez <eg...@apache.org>
> Sent: Monday, February 19, 2024 7:11 AM
> To: dev@kie.apache.org
> Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with data-index
> persistence addon
>
> Hi Francisco,
> To give you more context about this.
>
> STP is a concept, a process with certain constraints: no persistence and
> returning the outcome in the call (sync execution with no idle states). It
> was a requirement from a user in the past. One of the requirements was
> leaving no trail. In v7 was easy because you could disable the audit in
> that case. Actually we have the same way to do what we did in v7 in here as
> you can add/remove index just removing deps.
>
> We have the same outcome with different approaches and STP is already
> delivered.
>
> El lun, 19 feb 2024 a las 14:46, Francisco Javier Tirado Sarti (<
> ftirados@redhat.com>) escribió:
>
> > Regarding STP (which is not a concept that we have in the code. I mean
> STP
> > are processes as nonSTP are), I guess, as all processes, they were kept
> in
> > DataIndex once completed because users wanted (and still wants) to check
> > the result once the call had been performed. If we want to leave no trace
> > of them in DataIndex for some reason, we will need to make it a
> > Runtimes concept so DataIndex can handle them in a different way.
> >
> > On Mon, Feb 19, 2024 at 2:27 PM Enrique Gonzalez Martinez <
> > egonzalez@apache.org> wrote:
> >
> > > Alex:
> > > Right now the data index is working in the same way as it did in v7
> with
> > > the emitters. The only difference between two impl is that in here the
> > > storage is pgsql instead elastic search.  You are right regarding is a
> > > snapshot of the last state of the process but we did never define how
> > long
> > > would be alive that dats Honestly i am happy right now with the way it
> > > works. The clean up mechanism is still tbd because we still need to
> > discuss
> > > other stuff first.
> > >
> > >
> > > Regarding stp is to leave no trail because u can get the outcome
> directly
> > > from the call. It was defined like that in v7. So there is no use for
> the
> > > index or the audit.
> > >
> > > El lun, 19 feb 2024, 14:13, Francisco Javier Tirado Sarti <
> > > ftirados@redhat.com> escribió:
> > >
> > > > Hi Alex,
> > > > There has been some confusion about the purpose of DataIndex. To be
> > > honest
> > > > I believe they were already sorted out, but your e-mail makes me
> think
> > > that
> > > > is not the case ;). I let Kris to clarify that with you. My view is
> > that
> > > > data-index is a way to query recently closed and active processes
> (the
> > > key
> > > > here is the definition of recently, which in my opinion should be
> > > > configurable)
> > > > But, besides that discussion and being pragmatic, keeping finishing
> > > process
> > > > instances "for a while" in DataIndex was the only way for users to
> > query
> > > > the result of straight through processes. That's a function that
> cannot
> > > be
> > > > removed right now
> > > >
> > > > On Mon, Feb 19, 2024 at 1:33 PM Alex Porcelli <po...@apache.org>
> > > wrote:
> > > >
> > > > > if data index was supposed to provide snapshot view of the process
> > > > > instance… why do we keep it after the process instance is finished?
> > > > >
> > > > >
> > > > > On Mon, Feb 19, 2024 at 7:12 AM Francisco Javier Tirado Sarti <
> > > > > ftirados@redhat.com> wrote:
> > > > >
> > > > > > Hi Martin.
> > > > > > After taking a deeper look at this, I realize that the behaviour
> is
> > > the
> > > > > > expected one.
> > > > > > Runtimes DB does not track the completed process instance (that's
> > > what
> > > > > the
> > > > > > JDBCProcessInstances warn is telling us), but DataIndex, as
> > expected,
> > > > is
> > > > > > tracking it in processes and nodes table. And yes it will grow
> over
> > > > time.
> > > > > > What we need is some configurable purge mechanism for DataIndex,
> so
> > > it
> > > > > > eventually removes older completed process instances.
> > > > > >
> > > > > > On Tue, Feb 13, 2024 at 12:59 PM Francisco Javier Tirado Sarti <
> > > > > > ftirados@redhat.com> wrote:
> > > > > >
> > > > > > > Hi Martin,
> > > > > > > Good catch!. Looks like the skipping performed for process
> > > instances
> > > > is
> > > > > > > not applied to node instances. Something we definitely need to
> > > review
> > > > > on
> > > > > > > the runtimes side.
> > > > > > >
> > > > > > > On Mon, Feb 12, 2024 at 11:59 PM Martin Weiler
> > > > <mweiler@ibm.com.invalid
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> On a somewhat related note, testing a simple workflow (start
> ->
> > > > script
> > > > > > >> node -> end), I see the following messages in the logs:
> > > > > > >> 2024-02-12 22:49:50,493 28758dde544c WARN
> > > > > > >> [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> > > > > > >> (executor-thread-3) Skipping create of process instance id:
> > > > > > >> 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2
> > > > > > >>
> > > > > > >> So far, so good. And I'd expect to see no trace of this
> process
> > in
> > > > the
> > > > > > >> database if I don't have data audit enabled.
> > > > > > >>
> > > > > > >> However, the 'processes' table contains a row with state=2,
> with
> > > > > related
> > > > > > >> entries in the 'nodes' table. In a load test, I see these
> tables
> > > > grow
> > > > > > >> significantly over time. Am I missing something to have these
> > > > entries
> > > > > > >> cleaned up automatically?
> > > > > > >>
> > > > > > >> ________________________________________
> > > > > > >> From: Martin Weiler <mw...@ibm.com.INVALID>
> > > > > > >> Sent: Monday, February 12, 2024 3:40 PM
> > > > > > >> To: dev@kie.apache.org
> > > > > > >> Subject: [EXTERNAL] RE: [DISCUSSION] Performance issues with
> > > > > data-index
> > > > > > >> persistence addon
> > > > > > >>
> > > > > > >> Thanks everyone for your input. Based on this discussion, I
> > opened
> > > > the
> > > > > > >> following PR:
> > > > > > >> https://github.com/apache/incubator-kie-kogito-apps/pull/1985
> > > > > > >>
> > > > > > >> With this change, the performance seems to be stable over
> time:
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
> > > > > > >>
> > > > > > >> Martin
> > > > > > >>
> > > > > > >> ________________________________________
> > > > > > >> From: Gonzalo Muñoz <gm...@apache.org>
> > > > > > >> Sent: Friday, February 9, 2024 9:42 AM
> > > > > > >> To: dev@kie.apache.org
> > > > > > >> Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > > > > data-index
> > > > > > >> persistence addon
> > > > > > >>
> > > > > > >> Great work Francisco,
> > > > > > >> Martin, take a look at this link with some related tips (in
> case
> > > you
> > > > > > find
> > > > > > >> it useful):
> > > > > > >>
> https://www.cybertec-postgresql.com/en/index-your-foreign-key/
> > > > > > >>
> > > > > > >> El vie, 9 feb 2024 a las 17:20, Francisco Javier Tirado Sarti
> (<
> > > > > > >> ftirados@redhat.com>) escribió:
> > > > > > >>
> > > > > > >> > For the moment being, we will keep JPA till we exhaust all
> > > > > > >> possibilities,
> > > > > > >> > let's call switching from jpa to jdbc our hidden plan B ;)
> > > > > > >> > I already told Martin, but in order everyone to know, just
> > after
> > > > > > writing
> > > > > > >> > the previous email, I thought "what if Postgres is not
> > > > automatically
> > > > > > >> > indexing foreign keys like mysql?" and, eureka
> > > > > > >> > Postgres doc
> > > > > > >> >
> https://www.postgresql.org/docs/current/ddl-constraints.html
> > > > > > >> > Mysql doc
> > > > > > >> >
> > > > https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> > > > > > >> > These are the relevant excerpt
> > > > > > >> >
> > > > > > >> > *Postgresql*
> > > > > > >> > *A foreign key must reference columns that either are a
> > primary
> > > > key
> > > > > or
> > > > > > >> form
> > > > > > >> > a unique constraint, or are columns from a non-partial
> unique
> > > > index.
> > > > > > >> This
> > > > > > >> > means that the referenced columns always have an index to
> > allow
> > > > > > >> efficient
> > > > > > >> > lookups on whether a referencing row has a match. Since a
> > DELETE
> > > > of
> > > > > a
> > > > > > >> row
> > > > > > >> > from the referenced table or an UPDATE of a referenced
> column
> > > will
> > > > > > >> require
> > > > > > >> > a scan of the referencing table for rows matching the old
> > value,
> > > > it
> > > > > is
> > > > > > >> > often a good idea to index the referencing columns too.
> > Because
> > > > this
> > > > > > is
> > > > > > >> not
> > > > > > >> > always needed, and there are many choices available on how
> to
> > > > index,
> > > > > > the
> > > > > > >> > declaration of a foreign key constraint does not
> automatically
> > > > > create
> > > > > > an
> > > > > > >> > index on the referencing columns.*
> > > > > > >> > *Mysql*
> > > > > > >> > *MySQL requires that foreign key columns be indexed; if you
> > > > create a
> > > > > > >> table
> > > > > > >> > with a foreign key constraint but no index on a given
> column,
> > an
> > > > > index
> > > > > > >> is
> > > > > > >> > created. *
> > > > > > >> >
> > > > > > >> > So I asked Martin to especially create an index for
> > > > > > process_instance_id
> > > > > > >> > column on nodes table
> > > > > > >> > I think that will fix the problem detected on the thread
> dump.
> > > > > > >> > The simpler process test to verify queries are fine still
> > > stands,
> > > > > > >> though ;)
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <
> > > tzimanyi@apache.org
> > > > >
> > > > > > >> wrote:
> > > > > > >> >
> > > > > > >> > > I always preferred pure JDBC over Hibernate myself, just
> for
> > > the
> > > > > > sake
> > > > > > >> of
> > > > > > >> > > control of what is happening :) So I would not -1 that
> > myself.
> > > > > > >> > >
> > > > > > >> > > Tibor
> > > > > > >> > >
> > > > > > >> > > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado Sarti <
> > > > > > >> > > ftirados@redhat.com>
> > > > > > >> > > napísal(a):
> > > > > > >> > >
> > > > > > >> > > > Hi,
> > > > > > >> > > > Usually I do not want to talk about work in progress
> > because
> > > > > > >> > preliminary
> > > > > > >> > > > conclusions are pretty volatile but, well, there are a
> > > couple
> > > > of
> > > > > > >> things
> > > > > > >> > > > that can be concluded from the really valuable
> information
> > > > that
> > > > > > >> Martin
> > > > > > >> > > > provided.
> > > > > > >> > > > 1) In order to be able to determine if the number of
> > > > statements
> > > > > is
> > > > > > >> > larger
> > > > > > >> > > > than expected, I asked Martin to test with a simpler
> > process
> > > > > > >> > definition.
> > > > > > >> > > > One with just three nodes: start, script and end. The
> > script
> > > > one
> > > > > > >> should
> > > > > > >> > > > change just one variable. This way we can analyze if the
> > > > number
> > > > > of
> > > > > > >> > > queries
> > > > > > >> > > > is the expected one. From the single log (audit was
> > > activated
> > > > > > them)
> > > > > > >> my
> > > > > > >> > > > conclusion is that the number of insert/updates over
> > > processes
> > > > > and
> > > > > > >> > nodes
> > > > > > >> > > > (there a lot over task, that I will prefer to skip for
> > now,
> > > > baby
> > > > > > >> steps)
> > > > > > >> > > is
> > > > > > >> > > > the expected one.
> > > > > > >> > > > 2) Analysing the thread dump, we see around 15 threads
> > > > executing
> > > > > > >> this
> > > > > > >> > > line
> > > > > > >> > > > at
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > > > > >> > > > so its pretty clear the code to be optimized ;). I'm
> > > > evaluating
> > > > > > >> > > > possibilities within JPA/Hibernate, but I'm starting to
> > > think
> > > > > that
> > > > > > >> it
> > > > > > >> > > might
> > > > > > >> > > > be better to switch to JDBC and skip hibernate. Our
> lives
> > > will
> > > > > be
> > > > > > >> > > simpler,
> > > > > > >> > > > especially with a schema relatively simple like ours
> (that
> > > > will
> > > > > be
> > > > > > >> my
> > > > > > >> > > > recommendation if I was an external consultant)
> > > > > > >> > > >
> > > > > > >> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <
> > > > > tzimanyi@apache.org
> > > > > > >
> > > > > > >> > > wrote:
> > > > > > >> > > >
> > > > > > >> > > > > Hi,
> > > > > > >> > > > >
> > > > > > >> > > > > this will be a bit off-topic. However as far as
> > > > performance, I
> > > > > > >> think
> > > > > > >> > we
> > > > > > >> > > > > should think about that we have string primary keys
> > > (IDs). I
> > > > > > would
> > > > > > >> > > expect
> > > > > > >> > > > > the database systems are much better with indexing
> > numeric
> > > > > keys
> > > > > > >> than
> > > > > > >> > > > > strings. I remember from the past, when I was working
> > with
> > > > > DBs,
> > > > > > >> that
> > > > > > >> > > > using
> > > > > > >> > > > > strings as keys or indexes was a discouraged practice.
> > > > > > >> > > > >
> > > > > > >> > > > > Best regards,
> > > > > > >> > > > > Tibor
> > > > > > >> > > > >
> > > > > > >> > > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler
> > > > > <mweiler@ibm.com.invalid
> > > > > > >
> > > > > > >> > > > > napísal(a):
> > > > > > >> > > > >
> > > > > > >> > > > > > I changed the test to use MongoDB [1] and I don't
> see
> > a
> > > > > > >> performance
> > > > > > >> > > > > > degradation with this setup [2].
> > > > > > >> > > > > >
> > > > > > >> > > > > > Please keep us posted of your findings. Thanks!
> > > > > > >> > > > > >
> > > > > > >> > > > > > Martin
> > > > > > >> > > > > >
> > > > > > >> > > > > > [1]
> > > > > > >> > > > >
> > > > > > >> >
> > > > > >
> > > https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > > > >> > > > > > [2]
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > > > >> > > > > >
> > > > > > >> > > > > > ________________________________________
> > > > > > >> > > > > > From: Francisco Javier Tirado Sarti <
> > > ftirados@redhat.com>
> > > > > > >> > > > > > Sent: Wednesday, February 7, 2024 11:40 AM
> > > > > > >> > > > > > To: dev@kie.apache.org
> > > > > > >> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance
> > issues
> > > > with
> > > > > > >> > > data-index
> > > > > > >> > > > > > persistence addon
> > > > > > >> > > > > >
> > > > > > >> > > > > > yes, it can be index degradation because of size,
> but
> > I
> > > > > > believe
> > > > > > >> (I
> > > > > > >> > > > might
> > > > > > >> > > > > be
> > > > > > >> > > > > > wrong) the db is too small (yet) for that.
> > > > > > >> > > > > > But, eventually, Postgres, when the DB is huge
> enough,
> > > > > > >> unavoidably
> > > > > > >> > > will
> > > > > > >> > > > > > behave like the graphic that Martin sent.
> > > > > > >> > > > > > Since I believe we are not huge enough (yet), lets
> > rule
> > > > out
> > > > > > >> another
> > > > > > >> > > > issue
> > > > > > >> > > > > > by analysing the sql logs (I requested those to
> Martin
> > > > > offline
> > > > > > >> and
> > > > > > >> > he
> > > > > > >> > > > is
> > > > > > >> > > > > > going to kindly collect them).
> > > > > > >> > > > > > Also Im curious to know if Mongo behave in the same
> > way.
> > > > > > >> > > > > >
> > > > > > >> > > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez
> > > Martinez <
> > > > > > >> > > > > > egonzalez@apache.org> wrote:
> > > > > > >> > > > > >
> > > > > > >> > > > > > > Hi Francisco,
> > > > > > >> > > > > > > I would highly recommend to check indexes and how
> > the
> > > > > > updates
> > > > > > >> > work
> > > > > > >> > > in
> > > > > > >> > > > > > data
> > > > > > >> > > > > > > index to avoid full scan table and lock the full
> > > table.
> > > > > Some
> > > > > > >> db
> > > > > > >> > are
> > > > > > >> > > > > very
> > > > > > >> > > > > > > sensitive to that.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > El mié, 7 feb 2024, 18:41, Francisco Javier Tirado
> > > > Sarti <
> > > > > > >> > > > > > > ftirados@redhat.com> escribió:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > Hi Martin,
> > > > > > >> > > > > > > > While I analyze the data, let me ask you if it
> is
> > > > > possible
> > > > > > >> to
> > > > > > >> > > > perform
> > > > > > >> > > > > > > > another check (similar in a way to disabling
> > > > data-index
> > > > > > like
> > > > > > >> > you
> > > > > > >> > > > do)
> > > > > > >> > > > > > Can
> > > > > > >> > > > > > > > you switch to MongoDB persistence and check if
> the
> > > > same
> > > > > > >> > > degradation
> > > > > > >> > > > > > that
> > > > > > >> > > > > > > is
> > > > > > >> > > > > > > > there for postgres remains?
> > > > > > >> > > > > > > > I do not know if this is feasible but will
> > certainly
> > > > > > >> indicate
> > > > > > >> > the
> > > > > > >> > > > > > problem
> > > > > > >> > > > > > > > is on the postgres storage layer and I do not
> > have a
> > > > > clear
> > > > > > >> > > > prediction
> > > > > > >> > > > > > of
> > > > > > >> > > > > > > > what we will see when doing this switch.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler
> > > > > > >> > > > <mweiler@ibm.com.invalid
> > > > > > >> > > > > >
> > > > > > >> > > > > > > > wrote:
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > > Hi Francisco,
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > thanks for your work on this important topic!
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > I would like to share some test results here,
> > > which
> > > > > > might
> > > > > > >> > help
> > > > > > >> > > to
> > > > > > >> > > > > > > improve
> > > > > > >> > > > > > > > > the codebase even further. I am using the
> jmeter
> > > > based
> > > > > > >> test
> > > > > > >> > > case
> > > > > > >> > > > > from
> > > > > > >> > > > > > > > Pere
> > > > > > >> > > > > > > > > and Enrique (thanks guys!) [1] which uses a
> load
> > > of
> > > > 30
> > > > > > >> > threads
> > > > > > >> > > to
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > 1) start a new process instance (POST)
> > > > > > >> > > > > > > > > 2) retrieve tasks for a user (GET)
> > > > > > >> > > > > > > > > 3) fetches task details (GET)
> > > > > > >> > > > > > > > > 4) complete a task (POST)
> > > > > > >> > > > > > > > > 5) execute a query on data-audit
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > With this test setup, I noticed that the
> > > performance
> > > > > for
> > > > > > >> the
> > > > > > >> > > POST
> > > > > > >> > > > > > > > > requests, in particular the one to start a new
> > > > process
> > > > > > >> > > instance,
> > > > > > >> > > > > > > degrades
> > > > > > >> > > > > > > > > over time - see graph [2]. If I run the same
> > test
> > > > > > without
> > > > > > >> > > > > data-index,
> > > > > > >> > > > > > > > then
> > > > > > >> > > > > > > > > there is no such performance degradation [3].
> > You
> > > > can
> > > > > > >> find a
> > > > > > >> > > > thread
> > > > > > >> > > > > > > dump
> > > > > > >> > > > > > > > > captured a few minutes into the first test
> here
> > > [4]
> > > > > that
> > > > > > >> > might
> > > > > > >> > > > help
> > > > > > >> > > > > > to
> > > > > > >> > > > > > > > see
> > > > > > >> > > > > > > > > some of the contention points.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > I'd appreciate if you could take a look and
> see
> > if
> > > > > there
> > > > > > >> is
> > > > > > >> > > > > something
> > > > > > >> > > > > > > > that
> > > > > > >> > > > > > > > > can be further improved based on your previous
> > > work.
> > > > > If
> > > > > > >> you
> > > > > > >> > > need
> > > > > > >> > > > > any
> > > > > > >> > > > > > > > > additional data, let me know, but otherwise it
> > is
> > > > > > >> > > straightforward
> > > > > > >> > > > > to
> > > > > > >> > > > > > > run
> > > > > > >> > > > > > > > > the jmeter test as well.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Thanks,
> > > > > > >> > > > > > > > > Martin
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > [1]
> > > > > > >> https://github.com/pefernan/job-service-refactor-test/
> > > > > > >> > > > > > > > > [2]
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > > >> > > > > > > > > [3]
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > > >> > > > > > > > > [4]
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > ________________________________________
> > > > > > >> > > > > > > > > From: Francisco Javier Tirado Sarti <
> > > > > > ftirados@redhat.com>
> > > > > > >> > > > > > > > > Sent: Wednesday, January 17, 2024 9:13 AM
> > > > > > >> > > > > > > > > To: dev@kie.apache.org
> > > > > > >> > > > > > > > > Cc: Pere Fernandez Perez
> > > > > > >> > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION]
> Performance
> > > > > issues
> > > > > > >> with
> > > > > > >> > > > > > data-index
> > > > > > >> > > > > > > > > persistence addon
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Hi Alex,
> > > > > > >> > > > > > > > > I did not take times (which depends on a
> number
> > of
> > > > > > >> variables
> > > > > > >> > > that
> > > > > > >> > > > > > > > > drastically change between environments), but
> > > verify
> > > > > > that
> > > > > > >> the
> > > > > > >> > > > > number
> > > > > > >> > > > > > of
> > > > > > >> > > > > > > > > updates has been reduced drastically without
> > > losing
> > > > > > >> > > > functionality,
> > > > > > >> > > > > > > which
> > > > > > >> > > > > > > > is
> > > > > > >> > > > > > > > > objectively a good thing. If before the
> change,
> > > for
> > > > > > every
> > > > > > >> > node
> > > > > > >> > > > > > > executed,
> > > > > > >> > > > > > > > we
> > > > > > >> > > > > > > > > have an update for every node previously
> > executed,
> > > > so
> > > > > > if a
> > > > > > >> > > > process
> > > > > > >> > > > > > have
> > > > > > >> > > > > > > > 50
> > > > > > >> > > > > > > > > nodes to execute, we were performing nearly
> > > 50*51/2
> > > > > > >> updates,
> > > > > > >> > > > which
> > > > > > >> > > > > > > gives
> > > > > > >> > > > > > > > us
> > > > > > >> > > > > > > > > a total of  1275 updates, now we have just one
> > for
> > > > > every
> > > > > > >> node
> > > > > > >> > > > being
> > > > > > >> > > > > > > > > executed, implying a total of 50 updates.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli
> <
> > > > > > >> > > alex@porcelli.me>
> > > > > > >> > > > > > > wrote:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > Francisco,
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > I noticed that your PR has been merged, but
> I
> > > was
> > > > > > >> expecting
> > > > > > >> > > (at
> > > > > > >> > > > > > least
> > > > > > >> > > > > > > > > > was my understanding from this thread) that
> > > before
> > > > > > >> merging
> > > > > > >> > > some
> > > > > > >> > > > > > > > > > benchmark data would be shared in advance -
> to
> > > > > assess
> > > > > > >> the
> > > > > > >> > > > > > > cost/benefit
> > > > > > >> > > > > > > > > > of such a decent size change.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Do you have any information to share?
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM Francisco
> > Javier
> > > > > > Tirado
> > > > > > >> > Sarti
> > > > > > >> > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > Yes, as intended, now we have one select
> and
> > > one
> > > > > > >> > > > insert/update
> > > > > > >> > > > > > per
> > > > > > >> > > > > > > > node
> > > > > > >> > > > > > > > > > > event.
> > > > > > >> > > > > > > > > > > I moved the PR as ready for review and
> give
> > > > @Pere
> > > > > > >> > Fernandez
> > > > > > >> > > > > Perez
> > > > > > >> > > > > > > > > > > <pe...@redhat.com> permission to the
> > > branch
> > > > so
> > > > > > he
> > > > > > >> can
> > > > > > >> > > > edit
> > > > > > >> > > > > it
> > > > > > >> > > > > > > in
> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > > > > > > next two weeks (Ill be on PTO)  if
> desired,
> > > > before
> > > > > > >> > merging.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex
> > Porcelli
> > > <
> > > > > > >> > > > > alex@porcelli.me>
> > > > > > >> > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > Cool, thank you Francisco!
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > Did you manage to get some preliminary
> > data
> > > > > about
> > > > > > >> > > > > improvements?
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM
> Francisco
> > > > > Javier
> > > > > > >> > Tirado
> > > > > > >> > > > > Sarti
> > > > > > >> > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > Yes, after some delay because of
> > quarkus 3
> > > > > > >> migration.
> > > > > > >> > > Im
> > > > > > >> > > > > > > refining
> > > > > > >> > > > > > > > > > this
> > > > > > >> > > > > > > > > > > > > draft PR
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > >
> > > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex
> > > > Porcelli
> > > > > <
> > > > > > >> > > > > > > alex@porcelli.me>
> > > > > > >> > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > Any update or new findings on this
> > > topic?
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM
> > > Francisco
> > > > > > Javier
> > > > > > >> > > Tirado
> > > > > > >> > > > > > Sarti
> > > > > > >> > > > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > Hi Alex,
> > > > > > >> > > > > > > > > > > > > > > After considering different
> options
> > to
> > > > > > improve
> > > > > > >> > > > > > performance,
> > > > > > >> > > > > > > > we
> > > > > > >> > > > > > > > > > feel
> > > > > > >> > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > > it
> > > > > > >> > > > > > > > > > > > > > > is time to "partially" move away
> > from
> > > > the
> > > > > > >> current
> > > > > > >> > > Map
> > > > > > >> > > > > > style
> > > > > > >> > > > > > > > > > > > interface (
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > >> > > > > > > > > > > > > > )
> > > > > > >> > > > > > > > > > > > > > > which was shared with Trusty, to
> one
> > > > more
> > > > > > >> > suitable
> > > > > > >> > > > for
> > > > > > >> > > > > > > usage
> > > > > > >> > > > > > > > > > with a
> > > > > > >> > > > > > > > > > > > > > > relational DB like postgresql (but
> > > still
> > > > > > >> > compatible
> > > > > > >> > > > > with
> > > > > > >> > > > > > > big
> > > > > > >> > > > > > > > > > table
> > > > > > >> > > > > > > > > > > > dbs).
> > > > > > >> > > > > > > > > > > > > > > The idea will be to replace
> generic
> > > > > Storage
> > > > > > >> > > interface
> > > > > > >> > > > > by
> > > > > > >> > > > > > > four
> > > > > > >> > > > > > > > > > > > specific
> > > > > > >> > > > > > > > > > > > > > > interfaces (which will inherit
> from
> > a
> > > > > common
> > > > > > >> one
> > > > > > >> > > that
> > > > > > >> > > > > > keeps
> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > > > > > query
> > > > > > >> > > > > > > > > > > > > > part
> > > > > > >> > > > > > > > > > > > > > > at is it. with get and query
> > methods),
> > > > > that
> > > > > > >> will
> > > > > > >> > > > > include
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > > > required
> > > > > > >> > > > > > > > > > > > > > > modification operations for the
> four
> > > > > > DataIndex
> > > > > > >> > > > > "domains":
> > > > > > >> > > > > > > > > > > > > > processinstance,
> > > > > > >> > > > > > > > > > > > > > > usertask, processdefinitions and
> > jobs.
> > > > > Those
> > > > > > >> > > > interfaces
> > > > > > >> > > > > > > will
> > > > > > >> > > > > > > > > > define
> > > > > > >> > > > > > > > > > > > > > methods
> > > > > > >> > > > > > > > > > > > > > > like addNode, addVariable,
> > updateTask,
> > > > > > >> > > > > addAttachment.....
> > > > > > >> > > > > > > > that
> > > > > > >> > > > > > > > > > will
> > > > > > >> > > > > > > > > > > > allow
> > > > > > >> > > > > > > > > > > > > > > the persistent layer
> implementation
> > > to
> > > > > just
> > > > > > >> > update
> > > > > > >> > > > the
> > > > > > >> > > > > > > > needed
> > > > > > >> > > > > > > > > > info
> > > > > > >> > > > > > > > > > > > in
> > > > > > >> > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > DB  (for example, for addNode in
> > > > Postgres,
> > > > > > >> just
> > > > > > >> > > > insert
> > > > > > >> > > > > a
> > > > > > >> > > > > > > row
> > > > > > >> > > > > > > > > into
> > > > > > >> > > > > > > > > > > > nodes
> > > > > > >> > > > > > > > > > > > > > > table, for addNode in Mongo,
> > basically
> > > > the
> > > > > > >> same
> > > > > > >> > > > atomic
> > > > > > >> > > > > > > upsert
> > > > > > >> > > > > > > > > > > > operation
> > > > > > >> > > > > > > > > > > > > > > that is currently done).
> Therefore,
> > we
> > > > > > >> increase
> > > > > > >> > > > > > performance
> > > > > > >> > > > > > > > for
> > > > > > >> > > > > > > > > > > > Postgres
> > > > > > >> > > > > > > > > > > > > > > and keep the current one for
> Mongo.
> > > The
> > > > > > >> current
> > > > > > >> > DB
> > > > > > >> > > > > > schemas
> > > > > > >> > > > > > > > > won't
> > > > > > >> > > > > > > > > > be
> > > > > > >> > > > > > > > > > > > > > > touched.
> > > > > > >> > > > > > > > > > > > > > > Since the code change is large, I
> do
> > > not
> > > > > > think
> > > > > > >> > I'll
> > > > > > >> > > > be
> > > > > > >> > > > > > able
> > > > > > >> > > > > > > > to
> > > > > > >> > > > > > > > > > have
> > > > > > >> > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > PR
> > > > > > >> > > > > > > > > > > > > > > ready till next week.
> > > > > > >> > > > > > > > > > > > > > > But before starting, please let me
> > > know
> > > > if
> > > > > > >> that
> > > > > > >> > > > > approach
> > > > > > >> > > > > > is
> > > > > > >> > > > > > > > > fine
> > > > > > >> > > > > > > > > > for
> > > > > > >> > > > > > > > > > > > you.
> > > > > > >> > > > > > > > > > > > > > > Best regards.
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 6:55 PM
> Alex
> > > > > > Porcelli
> > > > > > >> <
> > > > > > >> > > > > > > > > alex@porcelli.me>
> > > > > > >> > > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > Thank you Francisco to getting
> > > deeper
> > > > on
> > > > > > >> this…
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > Looking forward to see the
> results
> > > of
> > > > > your
> > > > > > >> > > > suggested
> > > > > > >> > > > > > > > > > improvements.
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 9:40 AM
> > > > > Francisco
> > > > > > >> > Javier
> > > > > > >> > > > > Tirado
> > > > > > >> > > > > > > > > Sarti <
> > > > > > >> > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > I forgot to attach the queries
> > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at
> 3:04 PM
> > > > > > Francisco
> > > > > > >> > > Javier
> > > > > > >> > > > > > Tirado
> > > > > > >> > > > > > > > > > Sarti <
> > > > > > >> > > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > > >> Hi,
> > > > > > >> > > > > > > > > > > > > > > > >> A brief update on this topic.
> > > > > > >> > > > > > > > > > > > > > > > >> After doing a simple test
> with
> > > > > example
> > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > >> > > > > > > > > > > > > > > > ,
> > > > > > >> > > > > > > > > > > > > > > > >> the number of updates over
> > Nodes
> > > > > table
> > > > > > is
> > > > > > >> > n*n,
> > > > > > >> > > > so
> > > > > > >> > > > > we
> > > > > > >> > > > > > > > > manage
> > > > > > >> > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > obtain a
> > > > > > >> > > > > > > > > > > > > > > > >> perfect quadratic performance
> > > > > > >> degradation.
> > > > > > >> > The
> > > > > > >> > > > > > problem
> > > > > > >> > > > > > > > is
> > > > > > >> > > > > > > > > > worse
> > > > > > >> > > > > > > > > > > > in
> > > > > > >> > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > case
> > > > > > >> > > > > > > > > > > > > > > > >> of Serverless Workflow than
> in
> > > BPMN
> > > > > > >> because
> > > > > > >> > we
> > > > > > >> > > > the
> > > > > > >> > > > > > > > number
> > > > > > >> > > > > > > > > of
> > > > > > >> > > > > > > > > > > > nodes
> > > > > > >> > > > > > > > > > > > > > is
> > > > > > >> > > > > > > > > > > > > > > > >> greater than the number of
> > > states.
> > > > In
> > > > > > >> that
> > > > > > >> > > > > example N
> > > > > > >> > > > > > > is
> > > > > > >> > > > > > > > > 16,
> > > > > > >> > > > > > > > > > but
> > > > > > >> > > > > > > > > > > > for
> > > > > > >> > > > > > > > > > > > > > a
> > > > > > >> > > > > > > > > > > > > > > > more
> > > > > > >> > > > > > > > > > > > > > > > >> complex workflow it would be
> > > > > certainly
> > > > > > >> > large.
> > > > > > >> > > > > > > > > > > > > > > > >> I think that this is more
> > related
> > > > to
> > > > > > how
> > > > > > >> we
> > > > > > >> > > are
> > > > > > >> > > > > > > handling
> > > > > > >> > > > > > > > > > JPA in
> > > > > > >> > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > > > code,
> > > > > > >> > > > > > > > > > > > > > > > >> in particular the mapping
> from
> > > > model
> > > > > to
> > > > > > >> > entity
> > > > > > >> > > > > > > > (basically
> > > > > > >> > > > > > > > > > JPA is
> > > > > > >> > > > > > > > > > > > > > blind
> > > > > > >> > > > > > > > > > > > > > > > and
> > > > > > >> > > > > > > > > > > > > > > > >> has to update all nodes for
> > every
> > > > > write
> > > > > > >> > > because
> > > > > > >> > > > it
> > > > > > >> > > > > > > > > believes
> > > > > > >> > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > node has
> > > > > > >> > > > > > > > > > > > > > > > >> been updated, although it is
> > not)
> > > > > than
> > > > > > an
> > > > > > >> > > issue
> > > > > > >> > > > in
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > > > table
> > > > > > >> > > > > > > > > > > > > > definition.
> > > > > > >> > > > > > > > > > > > > > > > >> In fact, when using JPA,
> > > separating
> > > > > the
> > > > > > >> > server
> > > > > > >> > > > > model
> > > > > > >> > > > > > > > from
> > > > > > >> > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > JPA
> > > > > > >> > > > > > > > > > > > > > > > entity is
> > > > > > >> > > > > > > > > > > > > > > > >> not a good idea, especially
> if
> > > the
> > > > > > entity
> > > > > > >> > > > contains
> > > > > > >> > > > > > > > > > collections.
> > > > > > >> > > > > > > > > > > > I
> > > > > > >> > > > > > > > > > > > > > will
> > > > > > >> > > > > > > > > > > > > > > > try
> > > > > > >> > > > > > > > > > > > > > > > >> to change that without
> breaking
> > > > > > anything.
> > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > > > > > >> On Wed, Nov 22, 2023 at
> > 12:10 PM
> > > > > > Enrique
> > > > > > >> > > > Gonzalez
> > > > > > >> > > > > > > > > Martinez <
> > > > > > >> > > > > > > > > > > > > > > > >> egonzalez@apache.org> wrote:
> > > > > > >> > > > > > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > > > > > >>> After the events split you
> now
> > > > will
> > > > > > >> need to
> > > > > > >> > > > > create
> > > > > > >> > > > > > a
> > > > > > >> > > > > > > > node
> > > > > > >> > > > > > > > > > > > instance
> > > > > > >> > > > > > > > > > > > > > > > >>> model instance of making
> > > > independent
> > > > > > >> from
> > > > > > >> > the
> > > > > > >> > > > > > process
> > > > > > >> > > > > > > > > > instance.
> > > > > > >> > > > > > > > > > > > > > > > >>> That should do the trick.
> > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > > > >>> Regarding deleting/inserting
> > it
> > > > was
> > > > > > >> fixed
> > > > > > >> > at
> > > > > > >> > > > some
> > > > > > >> > > > > > > > point.
> > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > > > >>> El mar, 21 nov 2023 a las
> > 20:22,
> > > > > > >> Francisco
> > > > > > >> > > > Javier
> > > > > > >> > > > > > > > Tirado
> > > > > > >> > > > > > > > > > Sarti
> > > > > > >> > > > > > > > > > > > > > > > >>> (<ft...@redhat.com>)
> > > escribió:
> > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > >> > > > > > > > > > > > > > > > >>> > Hi Martin,
> > > > > > >> > > > > > > > > > > > > > > > >>> > I have a task to review
> > > > > performance
> > > > > > of
> > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > ProcessInstanceNodeDataEventMerger
> > > > > > >> > > > > > > > > > > > > > > > >>> > My idea is to reduce the
> > > number
> > > > of
> > > > > > >> delete
> > > > > > >> > > > > inserts
> > > > > > >> > > > > > > > when
> > > > > > >> > > > > > > > > > > > processing
> > > > > > >> > > > > > > > > > > > > > > > >>> events
> > > > > > >> > > > > > > > > > > > > > > > >>> > and try to do it
> > incremental.
> > > > > > >> > > > > > > > > > > > > > > > >>> > That should improve
> > > performance.
> > > > > > >> > > > > > > > > > > > > > > > >>> > PS:
> > > > > > >> > > > > > > > > > > > > > > > >>> > I was planning to send an
> > > e-mail
> > > > > > >> tomorrow
> > > > > > >> > > > > > > announcing
> > > > > > >> > > > > > > > > > that in
> > > > > > >> > > > > > > > > > > > > > case you
> > > > > > >> > > > > > > > > > > > > > > > >>> were
> > > > > > >> > > > > > > > > > > > > > > > >>> > already working on a fix
> for
> > > > > that. I
> > > > > > >> > assume
> > > > > > >> > > > you
> > > > > > >> > > > > > are
> > > > > > >> > > > > > > > not
> > > > > > >> > > > > > > > > > and I
> > > > > > >> > > > > > > > > > > > > > would
> > > > > > >> > > > > > > > > > > > > > > > be
> > > > > > >> > > > > > > > > > > > > > > > >>> > sending a PR soon.
> > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > >> > > > > > > > > > > > > > > > >>> > On Tue, Nov 21, 2023 at
> > > 6:09 PM
> > > > > > Martin
> > > > > > >> > > Weiler
> > > > > > >> > > > > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > >> > > > > > > > > > > > > > > > >>> > wrote:
> > > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > > >> > > > > > > > > > > > > > > > >>> > > I looked into the new
> > > examples
> > > > > > using
> > > > > > >> > > > > data-index
> > > > > > >> > > > > > > > > > persistence
> > > > > > >> > > > > > > > > > > > > > addon -
> > > > > > >> > > > > > > > > > > > > > > > >>> Neus'
> > > > > > >> > > > > > > > > > > > > > > > >>> > > PR#1813 [1] for
> serverless
> > > and
> > > > > > >> Pere's
> > > > > > >> > > > branch
> > > > > > >> > > > > > [2]
> > > > > > >> > > > > > > > for
> > > > > > >> > > > > > > > > > > > workflow
> > > > > > >> > > > > > > > > > > > > > > > (great
> > > > > > >> > > > > > > > > > > > > > > > >>> job
> > > > > > >> > > > > > > > > > > > > > > > >>> > > both!) - and they work
> > > without
> > > > > > >> issues
> > > > > > >> > > using
> > > > > > >> > > > > > > single
> > > > > > >> > > > > > > > > > > > requests.
> > > > > > >> > > > > > > > > > > > > > > > >>> However, under
> > > > > > >> > > > > > > > > > > > > > > > >>> > > some load (I used 'ab'
> for
> > > > > testing
> > > > > > >> > with a
> > > > > > >> > > > > light
> > > > > > >> > > > > > > > > > > > concurrency of
> > > > > > >> > > > > > > > > > > > > > 10
> > > > > > >> > > > > > > > > > > > > > > > >>> parallel
> > > > > > >> > > > > > > > > > > > > > > > >>> > > requests) I ran into the
> > > > > following
> > > > > > >> > > > problems:
> > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > >> > > > > > > > > > > > > > > > >>> > > (1) Large number of
> > > > > insert/delete
> > > > > > >> calls
> > > > > > >> > > > (eg.
> > > > > > >> > > > > > for
> > > > > > >> > > > > > > > > tables
> > > > > > >> > > > > > > > > > > > such as
> > > > > > >> > > > > > > > > > > > > > > > >>> nodes,
> > > > > > >> > > > > > > > > > > > > > > > >>> > > definitions, etc)
> > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > >> > > > > > > > > > > > > > > > >>> > > (2) Hibernate
> > > > > > >> OptimisticLockExceptions
> > > > > > >> > /
> > > > > > >> > > > > > > > > > > > StaleStateExceptions
> > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > >> > > > > > > > > > > > > > > > >>> > > (3) DB deadlocks
> > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > >> > > > > > > > > > > > > > > > >>> > > (4) Error responses,
> slow
> > > > > response
> > > > > > >> > times
> > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > >> > > > > > > > > > > > > > > > >>> > > The reason I am reaching
> > out
> > > > > with
> > > > > > >> this
> > > > > > >> > > > topic
> > > > > > >> > > > > > here
> > > > > > >> > > > > > > > is
> > > > > > >> > > > > > > > > to
> > > > > > >> > > > > > > > > > > > find
> > > > > > >> > > > > > > > > > > > > > out if
> > > > > > >> > > > > > > > > > > > > > > > >>> we are
> > > > > > >> > > > > > > > > > > > > > > > >>> > > aware of this issue, and
> > if
> > > > > > someone
> > > > > > >> is
> > > > > > >> > > > > already
> > > > > > >> > > > > > > > > looking
> > > > > > >> > > > > > > > > > > > into or
> > > > > > >> > > > > > > > > > > > > > > > being
> > > > > > >> > > > > > > > > > > > > > > > >>> > > assigned to it?
> > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > >> > > > > > > > > > > > > > > > >>> > > Thanks,
> > > > > > >> > > > > > > > > > > > > > > > >>> > > Martin
> > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > >> > > > > > > > > > > > > > > > >>> > > [1]
> > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> >
> > > https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > > >> > > > > > > > > > > > > > > > >>> > > [2]
> > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > >
> > > > > > >>
> > > > ---------------------------------------------------------------------
> > > > > > >> > > > > > > > > > > > > > > > >>> > > To unsubscribe, e-mail:
> > > > > > >> > > > > > > > > dev-unsubscribe@kie.apache.org
> > > > > > >> > > > > > > > > > > > > > > > >>> > > For additional commands,
> > > > e-mail:
> > > > > > >> > > > > > > > > > dev-help@kie.apache.org
> > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > >
> > > > > > >> >
> > > > >
> ---------------------------------------------------------------------
> > > > > > >> > > > > > > > > > > > > > > > >>> To unsubscribe, e-mail:
> > > > > > >> > > > > > > dev-unsubscribe@kie.apache.org
> > > > > > >> > > > > > > > > > > > > > > > >>> For additional commands,
> > e-mail:
> > > > > > >> > > > > > > > dev-help@kie.apache.org
> > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > >
> > > > > > >>
> > > > ---------------------------------------------------------------------
> > > > > > >> > > > > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > > > > >> > > > > > dev-unsubscribe@kie.apache.org
> > > > > > >> > > > > > > > > > > > > > > > > For additional commands,
> e-mail:
> > > > > > >> > > > > > > dev-help@kie.apache.org
> > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > >
> > > > > > >> >
> > > > >
> ---------------------------------------------------------------------
> > > > > > >> > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > > > > >> > > dev-unsubscribe@kie.apache.org
> > > > > > >> > > > > > > > > > > > > > For additional commands, e-mail:
> > > > > > >> > > > dev-help@kie.apache.org
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > >
> > > > > > >>
> > > > ---------------------------------------------------------------------
> > > > > > >> > > > > > > > > > > > To unsubscribe, e-mail:
> > > > > > >> dev-unsubscribe@kie.apache.org
> > > > > > >> > > > > > > > > > > > For additional commands, e-mail:
> > > > > > >> > dev-help@kie.apache.org
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > >
> > > > > > >> >
> > > > >
> ---------------------------------------------------------------------
> > > > > > >> > > > > > > > > > To unsubscribe, e-mail:
> > > > > > dev-unsubscribe@kie.apache.org
> > > > > > >> > > > > > > > > > For additional commands, e-mail:
> > > > > > >> dev-help@kie.apache.org
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > >
> > > > > > >>
> > > > ---------------------------------------------------------------------
> > > > > > >> > > > > > > > > To unsubscribe, e-mail:
> > > > > dev-unsubscribe@kie.apache.org
> > > > > > >> > > > > > > > > For additional commands, e-mail:
> > > > > > dev-help@kie.apache.org
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > ---------------------------------------------------------------------
> > > > > > >> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > >> For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> For additional commands, e-mail: dev-help@kie.apache.org
>
>

RE: [DISCUSSION] Performance issues with data-index persistence addon

[Martin Weiler <mw...@ibm.com.INVALID>]

IMO, it is good to have this discussion around data sanity now instead of putting it off until later when data has already accumulated in production environments.

Based on the input here, we are dealing with three types of data:
1. Runtime data - active instances only, engine cleans up the data automatically at process instance end
2. Historic log data - data created by data-audit intended for long term storage
3. Data-index data - somehow this data falls in between the two aforementioned categories, with the idea of the data being "recent", but not restricted to active instances only

We'd need purge strategies for both #2 and #3 (perhaps different ones, or with different config settings) in order to prevent unlimited data growth.

________________________________________
From: Enrique Gonzalez Martinez <eg...@apache.org>
Sent: Monday, February 19, 2024 7:11 AM
To: dev@kie.apache.org
Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with data-index persistence addon

Hi Francisco,
To give you more context about this.

STP is a concept, a process with certain constraints: no persistence and
returning the outcome in the call (sync execution with no idle states). It
was a requirement from a user in the past. One of the requirements was
leaving no trail. In v7 was easy because you could disable the audit in
that case. Actually we have the same way to do what we did in v7 in here as
you can add/remove index just removing deps.

We have the same outcome with different approaches and STP is already
delivered.

El lun, 19 feb 2024 a las 14:46, Francisco Javier Tirado Sarti (<
ftirados@redhat.com>) escribió:

> Regarding STP (which is not a concept that we have in the code. I mean STP
> are processes as nonSTP are), I guess, as all processes, they were kept in
> DataIndex once completed because users wanted (and still wants) to check
> the result once the call had been performed. If we want to leave no trace
> of them in DataIndex for some reason, we will need to make it a
> Runtimes concept so DataIndex can handle them in a different way.
>
> On Mon, Feb 19, 2024 at 2:27 PM Enrique Gonzalez Martinez <
> egonzalez@apache.org> wrote:
>
> > Alex:
> > Right now the data index is working in the same way as it did in v7 with
> > the emitters. The only difference between two impl is that in here the
> > storage is pgsql instead elastic search.  You are right regarding is a
> > snapshot of the last state of the process but we did never define how
> long
> > would be alive that dats Honestly i am happy right now with the way it
> > works. The clean up mechanism is still tbd because we still need to
> discuss
> > other stuff first.
> >
> >
> > Regarding stp is to leave no trail because u can get the outcome directly
> > from the call. It was defined like that in v7. So there is no use for the
> > index or the audit.
> >
> > El lun, 19 feb 2024, 14:13, Francisco Javier Tirado Sarti <
> > ftirados@redhat.com> escribió:
> >
> > > Hi Alex,
> > > There has been some confusion about the purpose of DataIndex. To be
> > honest
> > > I believe they were already sorted out, but your e-mail makes me think
> > that
> > > is not the case ;). I let Kris to clarify that with you. My view is
> that
> > > data-index is a way to query recently closed and active processes (the
> > key
> > > here is the definition of recently, which in my opinion should be
> > > configurable)
> > > But, besides that discussion and being pragmatic, keeping finishing
> > process
> > > instances "for a while" in DataIndex was the only way for users to
> query
> > > the result of straight through processes. That's a function that cannot
> > be
> > > removed right now
> > >
> > > On Mon, Feb 19, 2024 at 1:33 PM Alex Porcelli <po...@apache.org>
> > wrote:
> > >
> > > > if data index was supposed to provide snapshot view of the process
> > > > instance… why do we keep it after the process instance is finished?
> > > >
> > > >
> > > > On Mon, Feb 19, 2024 at 7:12 AM Francisco Javier Tirado Sarti <
> > > > ftirados@redhat.com> wrote:
> > > >
> > > > > Hi Martin.
> > > > > After taking a deeper look at this, I realize that the behaviour is
> > the
> > > > > expected one.
> > > > > Runtimes DB does not track the completed process instance (that's
> > what
> > > > the
> > > > > JDBCProcessInstances warn is telling us), but DataIndex, as
> expected,
> > > is
> > > > > tracking it in processes and nodes table. And yes it will grow over
> > > time.
> > > > > What we need is some configurable purge mechanism for DataIndex, so
> > it
> > > > > eventually removes older completed process instances.
> > > > >
> > > > > On Tue, Feb 13, 2024 at 12:59 PM Francisco Javier Tirado Sarti <
> > > > > ftirados@redhat.com> wrote:
> > > > >
> > > > > > Hi Martin,
> > > > > > Good catch!. Looks like the skipping performed for process
> > instances
> > > is
> > > > > > not applied to node instances. Something we definitely need to
> > review
> > > > on
> > > > > > the runtimes side.
> > > > > >
> > > > > > On Mon, Feb 12, 2024 at 11:59 PM Martin Weiler
> > > <mweiler@ibm.com.invalid
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> On a somewhat related note, testing a simple workflow (start ->
> > > script
> > > > > >> node -> end), I see the following messages in the logs:
> > > > > >> 2024-02-12 22:49:50,493 28758dde544c WARN
> > > > > >> [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> > > > > >> (executor-thread-3) Skipping create of process instance id:
> > > > > >> 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2
> > > > > >>
> > > > > >> So far, so good. And I'd expect to see no trace of this process
> in
> > > the
> > > > > >> database if I don't have data audit enabled.
> > > > > >>
> > > > > >> However, the 'processes' table contains a row with state=2, with
> > > > related
> > > > > >> entries in the 'nodes' table. In a load test, I see these tables
> > > grow
> > > > > >> significantly over time. Am I missing something to have these
> > > entries
> > > > > >> cleaned up automatically?
> > > > > >>
> > > > > >> ________________________________________
> > > > > >> From: Martin Weiler <mw...@ibm.com.INVALID>
> > > > > >> Sent: Monday, February 12, 2024 3:40 PM
> > > > > >> To: dev@kie.apache.org
> > > > > >> Subject: [EXTERNAL] RE: [DISCUSSION] Performance issues with
> > > > data-index
> > > > > >> persistence addon
> > > > > >>
> > > > > >> Thanks everyone for your input. Based on this discussion, I
> opened
> > > the
> > > > > >> following PR:
> > > > > >> https://github.com/apache/incubator-kie-kogito-apps/pull/1985
> > > > > >>
> > > > > >> With this change, the performance seems to be stable over time:
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
> > > > > >>
> > > > > >> Martin
> > > > > >>
> > > > > >> ________________________________________
> > > > > >> From: Gonzalo Muñoz <gm...@apache.org>
> > > > > >> Sent: Friday, February 9, 2024 9:42 AM
> > > > > >> To: dev@kie.apache.org
> > > > > >> Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > > > data-index
> > > > > >> persistence addon
> > > > > >>
> > > > > >> Great work Francisco,
> > > > > >> Martin, take a look at this link with some related tips (in case
> > you
> > > > > find
> > > > > >> it useful):
> > > > > >> https://www.cybertec-postgresql.com/en/index-your-foreign-key/
> > > > > >>
> > > > > >> El vie, 9 feb 2024 a las 17:20, Francisco Javier Tirado Sarti (<
> > > > > >> ftirados@redhat.com>) escribió:
> > > > > >>
> > > > > >> > For the moment being, we will keep JPA till we exhaust all
> > > > > >> possibilities,
> > > > > >> > let's call switching from jpa to jdbc our hidden plan B ;)
> > > > > >> > I already told Martin, but in order everyone to know, just
> after
> > > > > writing
> > > > > >> > the previous email, I thought "what if Postgres is not
> > > automatically
> > > > > >> > indexing foreign keys like mysql?" and, eureka
> > > > > >> > Postgres doc
> > > > > >> > https://www.postgresql.org/docs/current/ddl-constraints.html
> > > > > >> > Mysql doc
> > > > > >> >
> > > https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> > > > > >> > These are the relevant excerpt
> > > > > >> >
> > > > > >> > *Postgresql*
> > > > > >> > *A foreign key must reference columns that either are a
> primary
> > > key
> > > > or
> > > > > >> form
> > > > > >> > a unique constraint, or are columns from a non-partial unique
> > > index.
> > > > > >> This
> > > > > >> > means that the referenced columns always have an index to
> allow
> > > > > >> efficient
> > > > > >> > lookups on whether a referencing row has a match. Since a
> DELETE
> > > of
> > > > a
> > > > > >> row
> > > > > >> > from the referenced table or an UPDATE of a referenced column
> > will
> > > > > >> require
> > > > > >> > a scan of the referencing table for rows matching the old
> value,
> > > it
> > > > is
> > > > > >> > often a good idea to index the referencing columns too.
> Because
> > > this
> > > > > is
> > > > > >> not
> > > > > >> > always needed, and there are many choices available on how to
> > > index,
> > > > > the
> > > > > >> > declaration of a foreign key constraint does not automatically
> > > > create
> > > > > an
> > > > > >> > index on the referencing columns.*
> > > > > >> > *Mysql*
> > > > > >> > *MySQL requires that foreign key columns be indexed; if you
> > > create a
> > > > > >> table
> > > > > >> > with a foreign key constraint but no index on a given column,
> an
> > > > index
> > > > > >> is
> > > > > >> > created. *
> > > > > >> >
> > > > > >> > So I asked Martin to especially create an index for
> > > > > process_instance_id
> > > > > >> > column on nodes table
> > > > > >> > I think that will fix the problem detected on the thread dump.
> > > > > >> > The simpler process test to verify queries are fine still
> > stands,
> > > > > >> though ;)
> > > > > >> >
> > > > > >> >
> > > > > >> > On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <
> > tzimanyi@apache.org
> > > >
> > > > > >> wrote:
> > > > > >> >
> > > > > >> > > I always preferred pure JDBC over Hibernate myself, just for
> > the
> > > > > sake
> > > > > >> of
> > > > > >> > > control of what is happening :) So I would not -1 that
> myself.
> > > > > >> > >
> > > > > >> > > Tibor
> > > > > >> > >
> > > > > >> > > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado Sarti <
> > > > > >> > > ftirados@redhat.com>
> > > > > >> > > napísal(a):
> > > > > >> > >
> > > > > >> > > > Hi,
> > > > > >> > > > Usually I do not want to talk about work in progress
> because
> > > > > >> > preliminary
> > > > > >> > > > conclusions are pretty volatile but, well, there are a
> > couple
> > > of
> > > > > >> things
> > > > > >> > > > that can be concluded from the really valuable information
> > > that
> > > > > >> Martin
> > > > > >> > > > provided.
> > > > > >> > > > 1) In order to be able to determine if the number of
> > > statements
> > > > is
> > > > > >> > larger
> > > > > >> > > > than expected, I asked Martin to test with a simpler
> process
> > > > > >> > definition.
> > > > > >> > > > One with just three nodes: start, script and end. The
> script
> > > one
> > > > > >> should
> > > > > >> > > > change just one variable. This way we can analyze if the
> > > number
> > > > of
> > > > > >> > > queries
> > > > > >> > > > is the expected one. From the single log (audit was
> > activated
> > > > > them)
> > > > > >> my
> > > > > >> > > > conclusion is that the number of insert/updates over
> > processes
> > > > and
> > > > > >> > nodes
> > > > > >> > > > (there a lot over task, that I will prefer to skip for
> now,
> > > baby
> > > > > >> steps)
> > > > > >> > > is
> > > > > >> > > > the expected one.
> > > > > >> > > > 2) Analysing the thread dump, we see around 15 threads
> > > executing
> > > > > >> this
> > > > > >> > > line
> > > > > >> > > > at
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > > > >> > > > so its pretty clear the code to be optimized ;). I'm
> > > evaluating
> > > > > >> > > > possibilities within JPA/Hibernate, but I'm starting to
> > think
> > > > that
> > > > > >> it
> > > > > >> > > might
> > > > > >> > > > be better to switch to JDBC and skip hibernate. Our lives
> > will
> > > > be
> > > > > >> > > simpler,
> > > > > >> > > > especially with a schema relatively simple like ours (that
> > > will
> > > > be
> > > > > >> my
> > > > > >> > > > recommendation if I was an external consultant)
> > > > > >> > > >
> > > > > >> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <
> > > > tzimanyi@apache.org
> > > > > >
> > > > > >> > > wrote:
> > > > > >> > > >
> > > > > >> > > > > Hi,
> > > > > >> > > > >
> > > > > >> > > > > this will be a bit off-topic. However as far as
> > > performance, I
> > > > > >> think
> > > > > >> > we
> > > > > >> > > > > should think about that we have string primary keys
> > (IDs). I
> > > > > would
> > > > > >> > > expect
> > > > > >> > > > > the database systems are much better with indexing
> numeric
> > > > keys
> > > > > >> than
> > > > > >> > > > > strings. I remember from the past, when I was working
> with
> > > > DBs,
> > > > > >> that
> > > > > >> > > > using
> > > > > >> > > > > strings as keys or indexes was a discouraged practice.
> > > > > >> > > > >
> > > > > >> > > > > Best regards,
> > > > > >> > > > > Tibor
> > > > > >> > > > >
> > > > > >> > > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler
> > > > <mweiler@ibm.com.invalid
> > > > > >
> > > > > >> > > > > napísal(a):
> > > > > >> > > > >
> > > > > >> > > > > > I changed the test to use MongoDB [1] and I don't see
> a
> > > > > >> performance
> > > > > >> > > > > > degradation with this setup [2].
> > > > > >> > > > > >
> > > > > >> > > > > > Please keep us posted of your findings. Thanks!
> > > > > >> > > > > >
> > > > > >> > > > > > Martin
> > > > > >> > > > > >
> > > > > >> > > > > > [1]
> > > > > >> > > > >
> > > > > >> >
> > > > >
> > https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > > >> > > > > > [2]
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > > >> > > > > >
> > > > > >> > > > > > ________________________________________
> > > > > >> > > > > > From: Francisco Javier Tirado Sarti <
> > ftirados@redhat.com>
> > > > > >> > > > > > Sent: Wednesday, February 7, 2024 11:40 AM
> > > > > >> > > > > > To: dev@kie.apache.org
> > > > > >> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance
> issues
> > > with
> > > > > >> > > data-index
> > > > > >> > > > > > persistence addon
> > > > > >> > > > > >
> > > > > >> > > > > > yes, it can be index degradation because of size, but
> I
> > > > > believe
> > > > > >> (I
> > > > > >> > > > might
> > > > > >> > > > > be
> > > > > >> > > > > > wrong) the db is too small (yet) for that.
> > > > > >> > > > > > But, eventually, Postgres, when the DB is huge enough,
> > > > > >> unavoidably
> > > > > >> > > will
> > > > > >> > > > > > behave like the graphic that Martin sent.
> > > > > >> > > > > > Since I believe we are not huge enough (yet), lets
> rule
> > > out
> > > > > >> another
> > > > > >> > > > issue
> > > > > >> > > > > > by analysing the sql logs (I requested those to Martin
> > > > offline
> > > > > >> and
> > > > > >> > he
> > > > > >> > > > is
> > > > > >> > > > > > going to kindly collect them).
> > > > > >> > > > > > Also Im curious to know if Mongo behave in the same
> way.
> > > > > >> > > > > >
> > > > > >> > > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez
> > Martinez <
> > > > > >> > > > > > egonzalez@apache.org> wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > > Hi Francisco,
> > > > > >> > > > > > > I would highly recommend to check indexes and how
> the
> > > > > updates
> > > > > >> > work
> > > > > >> > > in
> > > > > >> > > > > > data
> > > > > >> > > > > > > index to avoid full scan table and lock the full
> > table.
> > > > Some
> > > > > >> db
> > > > > >> > are
> > > > > >> > > > > very
> > > > > >> > > > > > > sensitive to that.
> > > > > >> > > > > > >
> > > > > >> > > > > > > El mié, 7 feb 2024, 18:41, Francisco Javier Tirado
> > > Sarti <
> > > > > >> > > > > > > ftirados@redhat.com> escribió:
> > > > > >> > > > > > >
> > > > > >> > > > > > > > Hi Martin,
> > > > > >> > > > > > > > While I analyze the data, let me ask you if it is
> > > > possible
> > > > > >> to
> > > > > >> > > > perform
> > > > > >> > > > > > > > another check (similar in a way to disabling
> > > data-index
> > > > > like
> > > > > >> > you
> > > > > >> > > > do)
> > > > > >> > > > > > Can
> > > > > >> > > > > > > > you switch to MongoDB persistence and check if the
> > > same
> > > > > >> > > degradation
> > > > > >> > > > > > that
> > > > > >> > > > > > > is
> > > > > >> > > > > > > > there for postgres remains?
> > > > > >> > > > > > > > I do not know if this is feasible but will
> certainly
> > > > > >> indicate
> > > > > >> > the
> > > > > >> > > > > > problem
> > > > > >> > > > > > > > is on the postgres storage layer and I do not
> have a
> > > > clear
> > > > > >> > > > prediction
> > > > > >> > > > > > of
> > > > > >> > > > > > > > what we will see when doing this switch.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler
> > > > > >> > > > <mweiler@ibm.com.invalid
> > > > > >> > > > > >
> > > > > >> > > > > > > > wrote:
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > Hi Francisco,
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > thanks for your work on this important topic!
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > I would like to share some test results here,
> > which
> > > > > might
> > > > > >> > help
> > > > > >> > > to
> > > > > >> > > > > > > improve
> > > > > >> > > > > > > > > the codebase even further. I am using the jmeter
> > > based
> > > > > >> test
> > > > > >> > > case
> > > > > >> > > > > from
> > > > > >> > > > > > > > Pere
> > > > > >> > > > > > > > > and Enrique (thanks guys!) [1] which uses a load
> > of
> > > 30
> > > > > >> > threads
> > > > > >> > > to
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > 1) start a new process instance (POST)
> > > > > >> > > > > > > > > 2) retrieve tasks for a user (GET)
> > > > > >> > > > > > > > > 3) fetches task details (GET)
> > > > > >> > > > > > > > > 4) complete a task (POST)
> > > > > >> > > > > > > > > 5) execute a query on data-audit
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > With this test setup, I noticed that the
> > performance
> > > > for
> > > > > >> the
> > > > > >> > > POST
> > > > > >> > > > > > > > > requests, in particular the one to start a new
> > > process
> > > > > >> > > instance,
> > > > > >> > > > > > > degrades
> > > > > >> > > > > > > > > over time - see graph [2]. If I run the same
> test
> > > > > without
> > > > > >> > > > > data-index,
> > > > > >> > > > > > > > then
> > > > > >> > > > > > > > > there is no such performance degradation [3].
> You
> > > can
> > > > > >> find a
> > > > > >> > > > thread
> > > > > >> > > > > > > dump
> > > > > >> > > > > > > > > captured a few minutes into the first test here
> > [4]
> > > > that
> > > > > >> > might
> > > > > >> > > > help
> > > > > >> > > > > > to
> > > > > >> > > > > > > > see
> > > > > >> > > > > > > > > some of the contention points.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > I'd appreciate if you could take a look and see
> if
> > > > there
> > > > > >> is
> > > > > >> > > > > something
> > > > > >> > > > > > > > that
> > > > > >> > > > > > > > > can be further improved based on your previous
> > work.
> > > > If
> > > > > >> you
> > > > > >> > > need
> > > > > >> > > > > any
> > > > > >> > > > > > > > > additional data, let me know, but otherwise it
> is
> > > > > >> > > straightforward
> > > > > >> > > > > to
> > > > > >> > > > > > > run
> > > > > >> > > > > > > > > the jmeter test as well.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Thanks,
> > > > > >> > > > > > > > > Martin
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > [1]
> > > > > >> https://github.com/pefernan/job-service-refactor-test/
> > > > > >> > > > > > > > > [2]
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > >> > > > > > > > > [3]
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > >> > > > > > > > > [4]
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > ________________________________________
> > > > > >> > > > > > > > > From: Francisco Javier Tirado Sarti <
> > > > > ftirados@redhat.com>
> > > > > >> > > > > > > > > Sent: Wednesday, January 17, 2024 9:13 AM
> > > > > >> > > > > > > > > To: dev@kie.apache.org
> > > > > >> > > > > > > > > Cc: Pere Fernandez Perez
> > > > > >> > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance
> > > > issues
> > > > > >> with
> > > > > >> > > > > > data-index
> > > > > >> > > > > > > > > persistence addon
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Hi Alex,
> > > > > >> > > > > > > > > I did not take times (which depends on a number
> of
> > > > > >> variables
> > > > > >> > > that
> > > > > >> > > > > > > > > drastically change between environments), but
> > verify
> > > > > that
> > > > > >> the
> > > > > >> > > > > number
> > > > > >> > > > > > of
> > > > > >> > > > > > > > > updates has been reduced drastically without
> > losing
> > > > > >> > > > functionality,
> > > > > >> > > > > > > which
> > > > > >> > > > > > > > is
> > > > > >> > > > > > > > > objectively a good thing. If before the change,
> > for
> > > > > every
> > > > > >> > node
> > > > > >> > > > > > > executed,
> > > > > >> > > > > > > > we
> > > > > >> > > > > > > > > have an update for every node previously
> executed,
> > > so
> > > > > if a
> > > > > >> > > > process
> > > > > >> > > > > > have
> > > > > >> > > > > > > > 50
> > > > > >> > > > > > > > > nodes to execute, we were performing nearly
> > 50*51/2
> > > > > >> updates,
> > > > > >> > > > which
> > > > > >> > > > > > > gives
> > > > > >> > > > > > > > us
> > > > > >> > > > > > > > > a total of  1275 updates, now we have just one
> for
> > > > every
> > > > > >> node
> > > > > >> > > > being
> > > > > >> > > > > > > > > executed, implying a total of 50 updates.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli <
> > > > > >> > > alex@porcelli.me>
> > > > > >> > > > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > Francisco,
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > I noticed that your PR has been merged, but I
> > was
> > > > > >> expecting
> > > > > >> > > (at
> > > > > >> > > > > > least
> > > > > >> > > > > > > > > > was my understanding from this thread) that
> > before
> > > > > >> merging
> > > > > >> > > some
> > > > > >> > > > > > > > > > benchmark data would be shared in advance - to
> > > > assess
> > > > > >> the
> > > > > >> > > > > > > cost/benefit
> > > > > >> > > > > > > > > > of such a decent size change.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Do you have any information to share?
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM Francisco
> Javier
> > > > > Tirado
> > > > > >> > Sarti
> > > > > >> > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Yes, as intended, now we have one select and
> > one
> > > > > >> > > > insert/update
> > > > > >> > > > > > per
> > > > > >> > > > > > > > node
> > > > > >> > > > > > > > > > > event.
> > > > > >> > > > > > > > > > > I moved the PR as ready for review and give
> > > @Pere
> > > > > >> > Fernandez
> > > > > >> > > > > Perez
> > > > > >> > > > > > > > > > > <pe...@redhat.com> permission to the
> > branch
> > > so
> > > > > he
> > > > > >> can
> > > > > >> > > > edit
> > > > > >> > > > > it
> > > > > >> > > > > > > in
> > > > > >> > > > > > > > > the
> > > > > >> > > > > > > > > > > next two weeks (Ill be on PTO)  if desired,
> > > before
> > > > > >> > merging.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex
> Porcelli
> > <
> > > > > >> > > > > alex@porcelli.me>
> > > > > >> > > > > > > > > wrote:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Cool, thank you Francisco!
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Did you manage to get some preliminary
> data
> > > > about
> > > > > >> > > > > improvements?
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM Francisco
> > > > Javier
> > > > > >> > Tirado
> > > > > >> > > > > Sarti
> > > > > >> > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Yes, after some delay because of
> quarkus 3
> > > > > >> migration.
> > > > > >> > > Im
> > > > > >> > > > > > > refining
> > > > > >> > > > > > > > > > this
> > > > > >> > > > > > > > > > > > > draft PR
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > >
> > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex
> > > Porcelli
> > > > <
> > > > > >> > > > > > > alex@porcelli.me>
> > > > > >> > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Any update or new findings on this
> > topic?
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM
> > Francisco
> > > > > Javier
> > > > > >> > > Tirado
> > > > > >> > > > > > Sarti
> > > > > >> > > > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > Hi Alex,
> > > > > >> > > > > > > > > > > > > > > After considering different options
> to
> > > > > improve
> > > > > >> > > > > > performance,
> > > > > >> > > > > > > > we
> > > > > >> > > > > > > > > > feel
> > > > > >> > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > > it
> > > > > >> > > > > > > > > > > > > > > is time to "partially" move away
> from
> > > the
> > > > > >> current
> > > > > >> > > Map
> > > > > >> > > > > > style
> > > > > >> > > > > > > > > > > > interface (
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > >> > > > > > > > > > > > > > )
> > > > > >> > > > > > > > > > > > > > > which was shared with Trusty, to one
> > > more
> > > > > >> > suitable
> > > > > >> > > > for
> > > > > >> > > > > > > usage
> > > > > >> > > > > > > > > > with a
> > > > > >> > > > > > > > > > > > > > > relational DB like postgresql (but
> > still
> > > > > >> > compatible
> > > > > >> > > > > with
> > > > > >> > > > > > > big
> > > > > >> > > > > > > > > > table
> > > > > >> > > > > > > > > > > > dbs).
> > > > > >> > > > > > > > > > > > > > > The idea will be to replace generic
> > > > Storage
> > > > > >> > > interface
> > > > > >> > > > > by
> > > > > >> > > > > > > four
> > > > > >> > > > > > > > > > > > specific
> > > > > >> > > > > > > > > > > > > > > interfaces (which will inherit from
> a
> > > > common
> > > > > >> one
> > > > > >> > > that
> > > > > >> > > > > > keeps
> > > > > >> > > > > > > > the
> > > > > >> > > > > > > > > > query
> > > > > >> > > > > > > > > > > > > > part
> > > > > >> > > > > > > > > > > > > > > at is it. with get and query
> methods),
> > > > that
> > > > > >> will
> > > > > >> > > > > include
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > > > required
> > > > > >> > > > > > > > > > > > > > > modification operations for the four
> > > > > DataIndex
> > > > > >> > > > > "domains":
> > > > > >> > > > > > > > > > > > > > processinstance,
> > > > > >> > > > > > > > > > > > > > > usertask, processdefinitions and
> jobs.
> > > > Those
> > > > > >> > > > interfaces
> > > > > >> > > > > > > will
> > > > > >> > > > > > > > > > define
> > > > > >> > > > > > > > > > > > > > methods
> > > > > >> > > > > > > > > > > > > > > like addNode, addVariable,
> updateTask,
> > > > > >> > > > > addAttachment.....
> > > > > >> > > > > > > > that
> > > > > >> > > > > > > > > > will
> > > > > >> > > > > > > > > > > > allow
> > > > > >> > > > > > > > > > > > > > > the persistent layer implementation
> > to
> > > > just
> > > > > >> > update
> > > > > >> > > > the
> > > > > >> > > > > > > > needed
> > > > > >> > > > > > > > > > info
> > > > > >> > > > > > > > > > > > in
> > > > > >> > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > DB  (for example, for addNode in
> > > Postgres,
> > > > > >> just
> > > > > >> > > > insert
> > > > > >> > > > > a
> > > > > >> > > > > > > row
> > > > > >> > > > > > > > > into
> > > > > >> > > > > > > > > > > > nodes
> > > > > >> > > > > > > > > > > > > > > table, for addNode in Mongo,
> basically
> > > the
> > > > > >> same
> > > > > >> > > > atomic
> > > > > >> > > > > > > upsert
> > > > > >> > > > > > > > > > > > operation
> > > > > >> > > > > > > > > > > > > > > that is currently done). Therefore,
> we
> > > > > >> increase
> > > > > >> > > > > > performance
> > > > > >> > > > > > > > for
> > > > > >> > > > > > > > > > > > Postgres
> > > > > >> > > > > > > > > > > > > > > and keep the current one for Mongo.
> > The
> > > > > >> current
> > > > > >> > DB
> > > > > >> > > > > > schemas
> > > > > >> > > > > > > > > won't
> > > > > >> > > > > > > > > > be
> > > > > >> > > > > > > > > > > > > > > touched.
> > > > > >> > > > > > > > > > > > > > > Since the code change is large, I do
> > not
> > > > > think
> > > > > >> > I'll
> > > > > >> > > > be
> > > > > >> > > > > > able
> > > > > >> > > > > > > > to
> > > > > >> > > > > > > > > > have
> > > > > >> > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > PR
> > > > > >> > > > > > > > > > > > > > > ready till next week.
> > > > > >> > > > > > > > > > > > > > > But before starting, please let me
> > know
> > > if
> > > > > >> that
> > > > > >> > > > > approach
> > > > > >> > > > > > is
> > > > > >> > > > > > > > > fine
> > > > > >> > > > > > > > > > for
> > > > > >> > > > > > > > > > > > you.
> > > > > >> > > > > > > > > > > > > > > Best regards.
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 6:55 PM Alex
> > > > > Porcelli
> > > > > >> <
> > > > > >> > > > > > > > > alex@porcelli.me>
> > > > > >> > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > Thank you Francisco to getting
> > deeper
> > > on
> > > > > >> this…
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > Looking forward to see the results
> > of
> > > > your
> > > > > >> > > > suggested
> > > > > >> > > > > > > > > > improvements.
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 9:40 AM
> > > > Francisco
> > > > > >> > Javier
> > > > > >> > > > > Tirado
> > > > > >> > > > > > > > > Sarti <
> > > > > >> > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > I forgot to attach the queries
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 3:04 PM
> > > > > Francisco
> > > > > >> > > Javier
> > > > > >> > > > > > Tirado
> > > > > >> > > > > > > > > > Sarti <
> > > > > >> > > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > >> Hi,
> > > > > >> > > > > > > > > > > > > > > > >> A brief update on this topic.
> > > > > >> > > > > > > > > > > > > > > > >> After doing a simple test with
> > > > example
> > > > > >> > > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > >> > > > > > > > > > > > > > > > ,
> > > > > >> > > > > > > > > > > > > > > > >> the number of updates over
> Nodes
> > > > table
> > > > > is
> > > > > >> > n*n,
> > > > > >> > > > so
> > > > > >> > > > > we
> > > > > >> > > > > > > > > manage
> > > > > >> > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > obtain a
> > > > > >> > > > > > > > > > > > > > > > >> perfect quadratic performance
> > > > > >> degradation.
> > > > > >> > The
> > > > > >> > > > > > problem
> > > > > >> > > > > > > > is
> > > > > >> > > > > > > > > > worse
> > > > > >> > > > > > > > > > > > in
> > > > > >> > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > case
> > > > > >> > > > > > > > > > > > > > > > >> of Serverless Workflow than in
> > BPMN
> > > > > >> because
> > > > > >> > we
> > > > > >> > > > the
> > > > > >> > > > > > > > number
> > > > > >> > > > > > > > > of
> > > > > >> > > > > > > > > > > > nodes
> > > > > >> > > > > > > > > > > > > > is
> > > > > >> > > > > > > > > > > > > > > > >> greater than the number of
> > states.
> > > In
> > > > > >> that
> > > > > >> > > > > example N
> > > > > >> > > > > > > is
> > > > > >> > > > > > > > > 16,
> > > > > >> > > > > > > > > > but
> > > > > >> > > > > > > > > > > > for
> > > > > >> > > > > > > > > > > > > > a
> > > > > >> > > > > > > > > > > > > > > > more
> > > > > >> > > > > > > > > > > > > > > > >> complex workflow it would be
> > > > certainly
> > > > > >> > large.
> > > > > >> > > > > > > > > > > > > > > > >> I think that this is more
> related
> > > to
> > > > > how
> > > > > >> we
> > > > > >> > > are
> > > > > >> > > > > > > handling
> > > > > >> > > > > > > > > > JPA in
> > > > > >> > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > code,
> > > > > >> > > > > > > > > > > > > > > > >> in particular the mapping from
> > > model
> > > > to
> > > > > >> > entity
> > > > > >> > > > > > > > (basically
> > > > > >> > > > > > > > > > JPA is
> > > > > >> > > > > > > > > > > > > > blind
> > > > > >> > > > > > > > > > > > > > > > and
> > > > > >> > > > > > > > > > > > > > > > >> has to update all nodes for
> every
> > > > write
> > > > > >> > > because
> > > > > >> > > > it
> > > > > >> > > > > > > > > believes
> > > > > >> > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > node has
> > > > > >> > > > > > > > > > > > > > > > >> been updated, although it is
> not)
> > > > than
> > > > > an
> > > > > >> > > issue
> > > > > >> > > > in
> > > > > >> > > > > > the
> > > > > >> > > > > > > > > table
> > > > > >> > > > > > > > > > > > > > definition.
> > > > > >> > > > > > > > > > > > > > > > >> In fact, when using JPA,
> > separating
> > > > the
> > > > > >> > server
> > > > > >> > > > > model
> > > > > >> > > > > > > > from
> > > > > >> > > > > > > > > > the
> > > > > >> > > > > > > > > > > > JPA
> > > > > >> > > > > > > > > > > > > > > > entity is
> > > > > >> > > > > > > > > > > > > > > > >> not a good idea, especially if
> > the
> > > > > entity
> > > > > >> > > > contains
> > > > > >> > > > > > > > > > collections.
> > > > > >> > > > > > > > > > > > I
> > > > > >> > > > > > > > > > > > > > will
> > > > > >> > > > > > > > > > > > > > > > try
> > > > > >> > > > > > > > > > > > > > > > >> to change that without breaking
> > > > > anything.
> > > > > >> > > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > > > >> On Wed, Nov 22, 2023 at
> 12:10 PM
> > > > > Enrique
> > > > > >> > > > Gonzalez
> > > > > >> > > > > > > > > Martinez <
> > > > > >> > > > > > > > > > > > > > > > >> egonzalez@apache.org> wrote:
> > > > > >> > > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > > > >>> After the events split you now
> > > will
> > > > > >> need to
> > > > > >> > > > > create
> > > > > >> > > > > > a
> > > > > >> > > > > > > > node
> > > > > >> > > > > > > > > > > > instance
> > > > > >> > > > > > > > > > > > > > > > >>> model instance of making
> > > independent
> > > > > >> from
> > > > > >> > the
> > > > > >> > > > > > process
> > > > > >> > > > > > > > > > instance.
> > > > > >> > > > > > > > > > > > > > > > >>> That should do the trick.
> > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > > > >>> Regarding deleting/inserting
> it
> > > was
> > > > > >> fixed
> > > > > >> > at
> > > > > >> > > > some
> > > > > >> > > > > > > > point.
> > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > > > >>> El mar, 21 nov 2023 a las
> 20:22,
> > > > > >> Francisco
> > > > > >> > > > Javier
> > > > > >> > > > > > > > Tirado
> > > > > >> > > > > > > > > > Sarti
> > > > > >> > > > > > > > > > > > > > > > >>> (<ft...@redhat.com>)
> > escribió:
> > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > >> > > > > > > > > > > > > > > > >>> > Hi Martin,
> > > > > >> > > > > > > > > > > > > > > > >>> > I have a task to review
> > > > performance
> > > > > of
> > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > >> > > > > > > > > > > > > > > > >>> >
> > > ProcessInstanceNodeDataEventMerger
> > > > > >> > > > > > > > > > > > > > > > >>> > My idea is to reduce the
> > number
> > > of
> > > > > >> delete
> > > > > >> > > > > inserts
> > > > > >> > > > > > > > when
> > > > > >> > > > > > > > > > > > processing
> > > > > >> > > > > > > > > > > > > > > > >>> events
> > > > > >> > > > > > > > > > > > > > > > >>> > and try to do it
> incremental.
> > > > > >> > > > > > > > > > > > > > > > >>> > That should improve
> > performance.
> > > > > >> > > > > > > > > > > > > > > > >>> > PS:
> > > > > >> > > > > > > > > > > > > > > > >>> > I was planning to send an
> > e-mail
> > > > > >> tomorrow
> > > > > >> > > > > > > announcing
> > > > > >> > > > > > > > > > that in
> > > > > >> > > > > > > > > > > > > > case you
> > > > > >> > > > > > > > > > > > > > > > >>> were
> > > > > >> > > > > > > > > > > > > > > > >>> > already working on a fix for
> > > > that. I
> > > > > >> > assume
> > > > > >> > > > you
> > > > > >> > > > > > are
> > > > > >> > > > > > > > not
> > > > > >> > > > > > > > > > and I
> > > > > >> > > > > > > > > > > > > > would
> > > > > >> > > > > > > > > > > > > > > > be
> > > > > >> > > > > > > > > > > > > > > > >>> > sending a PR soon.
> > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > >> > > > > > > > > > > > > > > > >>> > On Tue, Nov 21, 2023 at
> > 6:09 PM
> > > > > Martin
> > > > > >> > > Weiler
> > > > > >> > > > > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > >> > > > > > > > > > > > > > > > >>> > wrote:
> > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > >> > > > > > > > > > > > > > > > >>> > > I looked into the new
> > examples
> > > > > using
> > > > > >> > > > > data-index
> > > > > >> > > > > > > > > > persistence
> > > > > >> > > > > > > > > > > > > > addon -
> > > > > >> > > > > > > > > > > > > > > > >>> Neus'
> > > > > >> > > > > > > > > > > > > > > > >>> > > PR#1813 [1] for serverless
> > and
> > > > > >> Pere's
> > > > > >> > > > branch
> > > > > >> > > > > > [2]
> > > > > >> > > > > > > > for
> > > > > >> > > > > > > > > > > > workflow
> > > > > >> > > > > > > > > > > > > > > > (great
> > > > > >> > > > > > > > > > > > > > > > >>> job
> > > > > >> > > > > > > > > > > > > > > > >>> > > both!) - and they work
> > without
> > > > > >> issues
> > > > > >> > > using
> > > > > >> > > > > > > single
> > > > > >> > > > > > > > > > > > requests.
> > > > > >> > > > > > > > > > > > > > > > >>> However, under
> > > > > >> > > > > > > > > > > > > > > > >>> > > some load (I used 'ab' for
> > > > testing
> > > > > >> > with a
> > > > > >> > > > > light
> > > > > >> > > > > > > > > > > > concurrency of
> > > > > >> > > > > > > > > > > > > > 10
> > > > > >> > > > > > > > > > > > > > > > >>> parallel
> > > > > >> > > > > > > > > > > > > > > > >>> > > requests) I ran into the
> > > > following
> > > > > >> > > > problems:
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > > (1) Large number of
> > > > insert/delete
> > > > > >> calls
> > > > > >> > > > (eg.
> > > > > >> > > > > > for
> > > > > >> > > > > > > > > tables
> > > > > >> > > > > > > > > > > > such as
> > > > > >> > > > > > > > > > > > > > > > >>> nodes,
> > > > > >> > > > > > > > > > > > > > > > >>> > > definitions, etc)
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > > (2) Hibernate
> > > > > >> OptimisticLockExceptions
> > > > > >> > /
> > > > > >> > > > > > > > > > > > StaleStateExceptions
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > > (3) DB deadlocks
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > > (4) Error responses, slow
> > > > response
> > > > > >> > times
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > > The reason I am reaching
> out
> > > > with
> > > > > >> this
> > > > > >> > > > topic
> > > > > >> > > > > > here
> > > > > >> > > > > > > > is
> > > > > >> > > > > > > > > to
> > > > > >> > > > > > > > > > > > find
> > > > > >> > > > > > > > > > > > > > out if
> > > > > >> > > > > > > > > > > > > > > > >>> we are
> > > > > >> > > > > > > > > > > > > > > > >>> > > aware of this issue, and
> if
> > > > > someone
> > > > > >> is
> > > > > >> > > > > already
> > > > > >> > > > > > > > > looking
> > > > > >> > > > > > > > > > > > into or
> > > > > >> > > > > > > > > > > > > > > > being
> > > > > >> > > > > > > > > > > > > > > > >>> > > assigned to it?
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > > Thanks,
> > > > > >> > > > > > > > > > > > > > > > >>> > > Martin
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > > [1]
> > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > >
> > > > > >> >
> > https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > >> > > > > > > > > > > > > > > > >>> > > [2]
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > >
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >> > > > > > > > > > > > > > > > >>> > > To unsubscribe, e-mail:
> > > > > >> > > > > > > > > dev-unsubscribe@kie.apache.org
> > > > > >> > > > > > > > > > > > > > > > >>> > > For additional commands,
> > > e-mail:
> > > > > >> > > > > > > > > > dev-help@kie.apache.org
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > >
> > > > > >> >
> > > > ---------------------------------------------------------------------
> > > > > >> > > > > > > > > > > > > > > > >>> To unsubscribe, e-mail:
> > > > > >> > > > > > > dev-unsubscribe@kie.apache.org
> > > > > >> > > > > > > > > > > > > > > > >>> For additional commands,
> e-mail:
> > > > > >> > > > > > > > dev-help@kie.apache.org
> > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > >
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >> > > > > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > > > >> > > > > > dev-unsubscribe@kie.apache.org
> > > > > >> > > > > > > > > > > > > > > > > For additional commands, e-mail:
> > > > > >> > > > > > > dev-help@kie.apache.org
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > >
> > > > > >> >
> > > > ---------------------------------------------------------------------
> > > > > >> > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > > > >> > > dev-unsubscribe@kie.apache.org
> > > > > >> > > > > > > > > > > > > > For additional commands, e-mail:
> > > > > >> > > > dev-help@kie.apache.org
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > >
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >> > > > > > > > > > > > To unsubscribe, e-mail:
> > > > > >> dev-unsubscribe@kie.apache.org
> > > > > >> > > > > > > > > > > > For additional commands, e-mail:
> > > > > >> > dev-help@kie.apache.org
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > >
> > > > > >> >
> > > > ---------------------------------------------------------------------
> > > > > >> > > > > > > > > > To unsubscribe, e-mail:
> > > > > dev-unsubscribe@kie.apache.org
> > > > > >> > > > > > > > > > For additional commands, e-mail:
> > > > > >> dev-help@kie.apache.org
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > >
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >> > > > > > > > > To unsubscribe, e-mail:
> > > > dev-unsubscribe@kie.apache.org
> > > > > >> > > > > > > > > For additional commands, e-mail:
> > > > > dev-help@kie.apache.org
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > >> For additional commands, e-mail: dev-help@kie.apache.org
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
For additional commands, e-mail: dev-help@kie.apache.org

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Enrique Gonzalez Martinez <eg...@apache.org>]

Hi Francisco,
To give you more context about this.

STP is a concept, a process with certain constraints: no persistence and
returning the outcome in the call (sync execution with no idle states). It
was a requirement from a user in the past. One of the requirements was
leaving no trail. In v7 was easy because you could disable the audit in
that case. Actually we have the same way to do what we did in v7 in here as
you can add/remove index just removing deps.

We have the same outcome with different approaches and STP is already
delivered.

El lun, 19 feb 2024 a las 14:46, Francisco Javier Tirado Sarti (<
ftirados@redhat.com>) escribió:

> Regarding STP (which is not a concept that we have in the code. I mean STP
> are processes as nonSTP are), I guess, as all processes, they were kept in
> DataIndex once completed because users wanted (and still wants) to check
> the result once the call had been performed. If we want to leave no trace
> of them in DataIndex for some reason, we will need to make it a
> Runtimes concept so DataIndex can handle them in a different way.
>
> On Mon, Feb 19, 2024 at 2:27 PM Enrique Gonzalez Martinez <
> egonzalez@apache.org> wrote:
>
> > Alex:
> > Right now the data index is working in the same way as it did in v7 with
> > the emitters. The only difference between two impl is that in here the
> > storage is pgsql instead elastic search.  You are right regarding is a
> > snapshot of the last state of the process but we did never define how
> long
> > would be alive that dats Honestly i am happy right now with the way it
> > works. The clean up mechanism is still tbd because we still need to
> discuss
> > other stuff first.
> >
> >
> > Regarding stp is to leave no trail because u can get the outcome directly
> > from the call. It was defined like that in v7. So there is no use for the
> > index or the audit.
> >
> > El lun, 19 feb 2024, 14:13, Francisco Javier Tirado Sarti <
> > ftirados@redhat.com> escribió:
> >
> > > Hi Alex,
> > > There has been some confusion about the purpose of DataIndex. To be
> > honest
> > > I believe they were already sorted out, but your e-mail makes me think
> > that
> > > is not the case ;). I let Kris to clarify that with you. My view is
> that
> > > data-index is a way to query recently closed and active processes (the
> > key
> > > here is the definition of recently, which in my opinion should be
> > > configurable)
> > > But, besides that discussion and being pragmatic, keeping finishing
> > process
> > > instances "for a while" in DataIndex was the only way for users to
> query
> > > the result of straight through processes. That's a function that cannot
> > be
> > > removed right now
> > >
> > > On Mon, Feb 19, 2024 at 1:33 PM Alex Porcelli <po...@apache.org>
> > wrote:
> > >
> > > > if data index was supposed to provide snapshot view of the process
> > > > instance… why do we keep it after the process instance is finished?
> > > >
> > > >
> > > > On Mon, Feb 19, 2024 at 7:12 AM Francisco Javier Tirado Sarti <
> > > > ftirados@redhat.com> wrote:
> > > >
> > > > > Hi Martin.
> > > > > After taking a deeper look at this, I realize that the behaviour is
> > the
> > > > > expected one.
> > > > > Runtimes DB does not track the completed process instance (that's
> > what
> > > > the
> > > > > JDBCProcessInstances warn is telling us), but DataIndex, as
> expected,
> > > is
> > > > > tracking it in processes and nodes table. And yes it will grow over
> > > time.
> > > > > What we need is some configurable purge mechanism for DataIndex, so
> > it
> > > > > eventually removes older completed process instances.
> > > > >
> > > > > On Tue, Feb 13, 2024 at 12:59 PM Francisco Javier Tirado Sarti <
> > > > > ftirados@redhat.com> wrote:
> > > > >
> > > > > > Hi Martin,
> > > > > > Good catch!. Looks like the skipping performed for process
> > instances
> > > is
> > > > > > not applied to node instances. Something we definitely need to
> > review
> > > > on
> > > > > > the runtimes side.
> > > > > >
> > > > > > On Mon, Feb 12, 2024 at 11:59 PM Martin Weiler
> > > <mweiler@ibm.com.invalid
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> On a somewhat related note, testing a simple workflow (start ->
> > > script
> > > > > >> node -> end), I see the following messages in the logs:
> > > > > >> 2024-02-12 22:49:50,493 28758dde544c WARN
> > > > > >> [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> > > > > >> (executor-thread-3) Skipping create of process instance id:
> > > > > >> 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2
> > > > > >>
> > > > > >> So far, so good. And I'd expect to see no trace of this process
> in
> > > the
> > > > > >> database if I don't have data audit enabled.
> > > > > >>
> > > > > >> However, the 'processes' table contains a row with state=2, with
> > > > related
> > > > > >> entries in the 'nodes' table. In a load test, I see these tables
> > > grow
> > > > > >> significantly over time. Am I missing something to have these
> > > entries
> > > > > >> cleaned up automatically?
> > > > > >>
> > > > > >> ________________________________________
> > > > > >> From: Martin Weiler <mw...@ibm.com.INVALID>
> > > > > >> Sent: Monday, February 12, 2024 3:40 PM
> > > > > >> To: dev@kie.apache.org
> > > > > >> Subject: [EXTERNAL] RE: [DISCUSSION] Performance issues with
> > > > data-index
> > > > > >> persistence addon
> > > > > >>
> > > > > >> Thanks everyone for your input. Based on this discussion, I
> opened
> > > the
> > > > > >> following PR:
> > > > > >> https://github.com/apache/incubator-kie-kogito-apps/pull/1985
> > > > > >>
> > > > > >> With this change, the performance seems to be stable over time:
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
> > > > > >>
> > > > > >> Martin
> > > > > >>
> > > > > >> ________________________________________
> > > > > >> From: Gonzalo Muñoz <gm...@apache.org>
> > > > > >> Sent: Friday, February 9, 2024 9:42 AM
> > > > > >> To: dev@kie.apache.org
> > > > > >> Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > > > data-index
> > > > > >> persistence addon
> > > > > >>
> > > > > >> Great work Francisco,
> > > > > >> Martin, take a look at this link with some related tips (in case
> > you
> > > > > find
> > > > > >> it useful):
> > > > > >> https://www.cybertec-postgresql.com/en/index-your-foreign-key/
> > > > > >>
> > > > > >> El vie, 9 feb 2024 a las 17:20, Francisco Javier Tirado Sarti (<
> > > > > >> ftirados@redhat.com>) escribió:
> > > > > >>
> > > > > >> > For the moment being, we will keep JPA till we exhaust all
> > > > > >> possibilities,
> > > > > >> > let's call switching from jpa to jdbc our hidden plan B ;)
> > > > > >> > I already told Martin, but in order everyone to know, just
> after
> > > > > writing
> > > > > >> > the previous email, I thought "what if Postgres is not
> > > automatically
> > > > > >> > indexing foreign keys like mysql?" and, eureka
> > > > > >> > Postgres doc
> > > > > >> > https://www.postgresql.org/docs/current/ddl-constraints.html
> > > > > >> > Mysql doc
> > > > > >> >
> > > https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> > > > > >> > These are the relevant excerpt
> > > > > >> >
> > > > > >> > *Postgresql*
> > > > > >> > *A foreign key must reference columns that either are a
> primary
> > > key
> > > > or
> > > > > >> form
> > > > > >> > a unique constraint, or are columns from a non-partial unique
> > > index.
> > > > > >> This
> > > > > >> > means that the referenced columns always have an index to
> allow
> > > > > >> efficient
> > > > > >> > lookups on whether a referencing row has a match. Since a
> DELETE
> > > of
> > > > a
> > > > > >> row
> > > > > >> > from the referenced table or an UPDATE of a referenced column
> > will
> > > > > >> require
> > > > > >> > a scan of the referencing table for rows matching the old
> value,
> > > it
> > > > is
> > > > > >> > often a good idea to index the referencing columns too.
> Because
> > > this
> > > > > is
> > > > > >> not
> > > > > >> > always needed, and there are many choices available on how to
> > > index,
> > > > > the
> > > > > >> > declaration of a foreign key constraint does not automatically
> > > > create
> > > > > an
> > > > > >> > index on the referencing columns.*
> > > > > >> > *Mysql*
> > > > > >> > *MySQL requires that foreign key columns be indexed; if you
> > > create a
> > > > > >> table
> > > > > >> > with a foreign key constraint but no index on a given column,
> an
> > > > index
> > > > > >> is
> > > > > >> > created. *
> > > > > >> >
> > > > > >> > So I asked Martin to especially create an index for
> > > > > process_instance_id
> > > > > >> > column on nodes table
> > > > > >> > I think that will fix the problem detected on the thread dump.
> > > > > >> > The simpler process test to verify queries are fine still
> > stands,
> > > > > >> though ;)
> > > > > >> >
> > > > > >> >
> > > > > >> > On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <
> > tzimanyi@apache.org
> > > >
> > > > > >> wrote:
> > > > > >> >
> > > > > >> > > I always preferred pure JDBC over Hibernate myself, just for
> > the
> > > > > sake
> > > > > >> of
> > > > > >> > > control of what is happening :) So I would not -1 that
> myself.
> > > > > >> > >
> > > > > >> > > Tibor
> > > > > >> > >
> > > > > >> > > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado Sarti <
> > > > > >> > > ftirados@redhat.com>
> > > > > >> > > napísal(a):
> > > > > >> > >
> > > > > >> > > > Hi,
> > > > > >> > > > Usually I do not want to talk about work in progress
> because
> > > > > >> > preliminary
> > > > > >> > > > conclusions are pretty volatile but, well, there are a
> > couple
> > > of
> > > > > >> things
> > > > > >> > > > that can be concluded from the really valuable information
> > > that
> > > > > >> Martin
> > > > > >> > > > provided.
> > > > > >> > > > 1) In order to be able to determine if the number of
> > > statements
> > > > is
> > > > > >> > larger
> > > > > >> > > > than expected, I asked Martin to test with a simpler
> process
> > > > > >> > definition.
> > > > > >> > > > One with just three nodes: start, script and end. The
> script
> > > one
> > > > > >> should
> > > > > >> > > > change just one variable. This way we can analyze if the
> > > number
> > > > of
> > > > > >> > > queries
> > > > > >> > > > is the expected one. From the single log (audit was
> > activated
> > > > > them)
> > > > > >> my
> > > > > >> > > > conclusion is that the number of insert/updates over
> > processes
> > > > and
> > > > > >> > nodes
> > > > > >> > > > (there a lot over task, that I will prefer to skip for
> now,
> > > baby
> > > > > >> steps)
> > > > > >> > > is
> > > > > >> > > > the expected one.
> > > > > >> > > > 2) Analysing the thread dump, we see around 15 threads
> > > executing
> > > > > >> this
> > > > > >> > > line
> > > > > >> > > > at
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > > > >> > > > so its pretty clear the code to be optimized ;). I'm
> > > evaluating
> > > > > >> > > > possibilities within JPA/Hibernate, but I'm starting to
> > think
> > > > that
> > > > > >> it
> > > > > >> > > might
> > > > > >> > > > be better to switch to JDBC and skip hibernate. Our lives
> > will
> > > > be
> > > > > >> > > simpler,
> > > > > >> > > > especially with a schema relatively simple like ours (that
> > > will
> > > > be
> > > > > >> my
> > > > > >> > > > recommendation if I was an external consultant)
> > > > > >> > > >
> > > > > >> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <
> > > > tzimanyi@apache.org
> > > > > >
> > > > > >> > > wrote:
> > > > > >> > > >
> > > > > >> > > > > Hi,
> > > > > >> > > > >
> > > > > >> > > > > this will be a bit off-topic. However as far as
> > > performance, I
> > > > > >> think
> > > > > >> > we
> > > > > >> > > > > should think about that we have string primary keys
> > (IDs). I
> > > > > would
> > > > > >> > > expect
> > > > > >> > > > > the database systems are much better with indexing
> numeric
> > > > keys
> > > > > >> than
> > > > > >> > > > > strings. I remember from the past, when I was working
> with
> > > > DBs,
> > > > > >> that
> > > > > >> > > > using
> > > > > >> > > > > strings as keys or indexes was a discouraged practice.
> > > > > >> > > > >
> > > > > >> > > > > Best regards,
> > > > > >> > > > > Tibor
> > > > > >> > > > >
> > > > > >> > > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler
> > > > <mweiler@ibm.com.invalid
> > > > > >
> > > > > >> > > > > napísal(a):
> > > > > >> > > > >
> > > > > >> > > > > > I changed the test to use MongoDB [1] and I don't see
> a
> > > > > >> performance
> > > > > >> > > > > > degradation with this setup [2].
> > > > > >> > > > > >
> > > > > >> > > > > > Please keep us posted of your findings. Thanks!
> > > > > >> > > > > >
> > > > > >> > > > > > Martin
> > > > > >> > > > > >
> > > > > >> > > > > > [1]
> > > > > >> > > > >
> > > > > >> >
> > > > >
> > https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > > >> > > > > > [2]
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > > >> > > > > >
> > > > > >> > > > > > ________________________________________
> > > > > >> > > > > > From: Francisco Javier Tirado Sarti <
> > ftirados@redhat.com>
> > > > > >> > > > > > Sent: Wednesday, February 7, 2024 11:40 AM
> > > > > >> > > > > > To: dev@kie.apache.org
> > > > > >> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance
> issues
> > > with
> > > > > >> > > data-index
> > > > > >> > > > > > persistence addon
> > > > > >> > > > > >
> > > > > >> > > > > > yes, it can be index degradation because of size, but
> I
> > > > > believe
> > > > > >> (I
> > > > > >> > > > might
> > > > > >> > > > > be
> > > > > >> > > > > > wrong) the db is too small (yet) for that.
> > > > > >> > > > > > But, eventually, Postgres, when the DB is huge enough,
> > > > > >> unavoidably
> > > > > >> > > will
> > > > > >> > > > > > behave like the graphic that Martin sent.
> > > > > >> > > > > > Since I believe we are not huge enough (yet), lets
> rule
> > > out
> > > > > >> another
> > > > > >> > > > issue
> > > > > >> > > > > > by analysing the sql logs (I requested those to Martin
> > > > offline
> > > > > >> and
> > > > > >> > he
> > > > > >> > > > is
> > > > > >> > > > > > going to kindly collect them).
> > > > > >> > > > > > Also Im curious to know if Mongo behave in the same
> way.
> > > > > >> > > > > >
> > > > > >> > > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez
> > Martinez <
> > > > > >> > > > > > egonzalez@apache.org> wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > > Hi Francisco,
> > > > > >> > > > > > > I would highly recommend to check indexes and how
> the
> > > > > updates
> > > > > >> > work
> > > > > >> > > in
> > > > > >> > > > > > data
> > > > > >> > > > > > > index to avoid full scan table and lock the full
> > table.
> > > > Some
> > > > > >> db
> > > > > >> > are
> > > > > >> > > > > very
> > > > > >> > > > > > > sensitive to that.
> > > > > >> > > > > > >
> > > > > >> > > > > > > El mié, 7 feb 2024, 18:41, Francisco Javier Tirado
> > > Sarti <
> > > > > >> > > > > > > ftirados@redhat.com> escribió:
> > > > > >> > > > > > >
> > > > > >> > > > > > > > Hi Martin,
> > > > > >> > > > > > > > While I analyze the data, let me ask you if it is
> > > > possible
> > > > > >> to
> > > > > >> > > > perform
> > > > > >> > > > > > > > another check (similar in a way to disabling
> > > data-index
> > > > > like
> > > > > >> > you
> > > > > >> > > > do)
> > > > > >> > > > > > Can
> > > > > >> > > > > > > > you switch to MongoDB persistence and check if the
> > > same
> > > > > >> > > degradation
> > > > > >> > > > > > that
> > > > > >> > > > > > > is
> > > > > >> > > > > > > > there for postgres remains?
> > > > > >> > > > > > > > I do not know if this is feasible but will
> certainly
> > > > > >> indicate
> > > > > >> > the
> > > > > >> > > > > > problem
> > > > > >> > > > > > > > is on the postgres storage layer and I do not
> have a
> > > > clear
> > > > > >> > > > prediction
> > > > > >> > > > > > of
> > > > > >> > > > > > > > what we will see when doing this switch.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler
> > > > > >> > > > <mweiler@ibm.com.invalid
> > > > > >> > > > > >
> > > > > >> > > > > > > > wrote:
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > Hi Francisco,
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > thanks for your work on this important topic!
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > I would like to share some test results here,
> > which
> > > > > might
> > > > > >> > help
> > > > > >> > > to
> > > > > >> > > > > > > improve
> > > > > >> > > > > > > > > the codebase even further. I am using the jmeter
> > > based
> > > > > >> test
> > > > > >> > > case
> > > > > >> > > > > from
> > > > > >> > > > > > > > Pere
> > > > > >> > > > > > > > > and Enrique (thanks guys!) [1] which uses a load
> > of
> > > 30
> > > > > >> > threads
> > > > > >> > > to
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > 1) start a new process instance (POST)
> > > > > >> > > > > > > > > 2) retrieve tasks for a user (GET)
> > > > > >> > > > > > > > > 3) fetches task details (GET)
> > > > > >> > > > > > > > > 4) complete a task (POST)
> > > > > >> > > > > > > > > 5) execute a query on data-audit
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > With this test setup, I noticed that the
> > performance
> > > > for
> > > > > >> the
> > > > > >> > > POST
> > > > > >> > > > > > > > > requests, in particular the one to start a new
> > > process
> > > > > >> > > instance,
> > > > > >> > > > > > > degrades
> > > > > >> > > > > > > > > over time - see graph [2]. If I run the same
> test
> > > > > without
> > > > > >> > > > > data-index,
> > > > > >> > > > > > > > then
> > > > > >> > > > > > > > > there is no such performance degradation [3].
> You
> > > can
> > > > > >> find a
> > > > > >> > > > thread
> > > > > >> > > > > > > dump
> > > > > >> > > > > > > > > captured a few minutes into the first test here
> > [4]
> > > > that
> > > > > >> > might
> > > > > >> > > > help
> > > > > >> > > > > > to
> > > > > >> > > > > > > > see
> > > > > >> > > > > > > > > some of the contention points.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > I'd appreciate if you could take a look and see
> if
> > > > there
> > > > > >> is
> > > > > >> > > > > something
> > > > > >> > > > > > > > that
> > > > > >> > > > > > > > > can be further improved based on your previous
> > work.
> > > > If
> > > > > >> you
> > > > > >> > > need
> > > > > >> > > > > any
> > > > > >> > > > > > > > > additional data, let me know, but otherwise it
> is
> > > > > >> > > straightforward
> > > > > >> > > > > to
> > > > > >> > > > > > > run
> > > > > >> > > > > > > > > the jmeter test as well.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Thanks,
> > > > > >> > > > > > > > > Martin
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > [1]
> > > > > >> https://github.com/pefernan/job-service-refactor-test/
> > > > > >> > > > > > > > > [2]
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > >> > > > > > > > > [3]
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > >> > > > > > > > > [4]
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > ________________________________________
> > > > > >> > > > > > > > > From: Francisco Javier Tirado Sarti <
> > > > > ftirados@redhat.com>
> > > > > >> > > > > > > > > Sent: Wednesday, January 17, 2024 9:13 AM
> > > > > >> > > > > > > > > To: dev@kie.apache.org
> > > > > >> > > > > > > > > Cc: Pere Fernandez Perez
> > > > > >> > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance
> > > > issues
> > > > > >> with
> > > > > >> > > > > > data-index
> > > > > >> > > > > > > > > persistence addon
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Hi Alex,
> > > > > >> > > > > > > > > I did not take times (which depends on a number
> of
> > > > > >> variables
> > > > > >> > > that
> > > > > >> > > > > > > > > drastically change between environments), but
> > verify
> > > > > that
> > > > > >> the
> > > > > >> > > > > number
> > > > > >> > > > > > of
> > > > > >> > > > > > > > > updates has been reduced drastically without
> > losing
> > > > > >> > > > functionality,
> > > > > >> > > > > > > which
> > > > > >> > > > > > > > is
> > > > > >> > > > > > > > > objectively a good thing. If before the change,
> > for
> > > > > every
> > > > > >> > node
> > > > > >> > > > > > > executed,
> > > > > >> > > > > > > > we
> > > > > >> > > > > > > > > have an update for every node previously
> executed,
> > > so
> > > > > if a
> > > > > >> > > > process
> > > > > >> > > > > > have
> > > > > >> > > > > > > > 50
> > > > > >> > > > > > > > > nodes to execute, we were performing nearly
> > 50*51/2
> > > > > >> updates,
> > > > > >> > > > which
> > > > > >> > > > > > > gives
> > > > > >> > > > > > > > us
> > > > > >> > > > > > > > > a total of  1275 updates, now we have just one
> for
> > > > every
> > > > > >> node
> > > > > >> > > > being
> > > > > >> > > > > > > > > executed, implying a total of 50 updates.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli <
> > > > > >> > > alex@porcelli.me>
> > > > > >> > > > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > Francisco,
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > I noticed that your PR has been merged, but I
> > was
> > > > > >> expecting
> > > > > >> > > (at
> > > > > >> > > > > > least
> > > > > >> > > > > > > > > > was my understanding from this thread) that
> > before
> > > > > >> merging
> > > > > >> > > some
> > > > > >> > > > > > > > > > benchmark data would be shared in advance - to
> > > > assess
> > > > > >> the
> > > > > >> > > > > > > cost/benefit
> > > > > >> > > > > > > > > > of such a decent size change.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Do you have any information to share?
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM Francisco
> Javier
> > > > > Tirado
> > > > > >> > Sarti
> > > > > >> > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Yes, as intended, now we have one select and
> > one
> > > > > >> > > > insert/update
> > > > > >> > > > > > per
> > > > > >> > > > > > > > node
> > > > > >> > > > > > > > > > > event.
> > > > > >> > > > > > > > > > > I moved the PR as ready for review and give
> > > @Pere
> > > > > >> > Fernandez
> > > > > >> > > > > Perez
> > > > > >> > > > > > > > > > > <pe...@redhat.com> permission to the
> > branch
> > > so
> > > > > he
> > > > > >> can
> > > > > >> > > > edit
> > > > > >> > > > > it
> > > > > >> > > > > > > in
> > > > > >> > > > > > > > > the
> > > > > >> > > > > > > > > > > next two weeks (Ill be on PTO)  if desired,
> > > before
> > > > > >> > merging.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex
> Porcelli
> > <
> > > > > >> > > > > alex@porcelli.me>
> > > > > >> > > > > > > > > wrote:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Cool, thank you Francisco!
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Did you manage to get some preliminary
> data
> > > > about
> > > > > >> > > > > improvements?
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM Francisco
> > > > Javier
> > > > > >> > Tirado
> > > > > >> > > > > Sarti
> > > > > >> > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Yes, after some delay because of
> quarkus 3
> > > > > >> migration.
> > > > > >> > > Im
> > > > > >> > > > > > > refining
> > > > > >> > > > > > > > > > this
> > > > > >> > > > > > > > > > > > > draft PR
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > >
> > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex
> > > Porcelli
> > > > <
> > > > > >> > > > > > > alex@porcelli.me>
> > > > > >> > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Any update or new findings on this
> > topic?
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM
> > Francisco
> > > > > Javier
> > > > > >> > > Tirado
> > > > > >> > > > > > Sarti
> > > > > >> > > > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > Hi Alex,
> > > > > >> > > > > > > > > > > > > > > After considering different options
> to
> > > > > improve
> > > > > >> > > > > > performance,
> > > > > >> > > > > > > > we
> > > > > >> > > > > > > > > > feel
> > > > > >> > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > > it
> > > > > >> > > > > > > > > > > > > > > is time to "partially" move away
> from
> > > the
> > > > > >> current
> > > > > >> > > Map
> > > > > >> > > > > > style
> > > > > >> > > > > > > > > > > > interface (
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > >> > > > > > > > > > > > > > )
> > > > > >> > > > > > > > > > > > > > > which was shared with Trusty, to one
> > > more
> > > > > >> > suitable
> > > > > >> > > > for
> > > > > >> > > > > > > usage
> > > > > >> > > > > > > > > > with a
> > > > > >> > > > > > > > > > > > > > > relational DB like postgresql (but
> > still
> > > > > >> > compatible
> > > > > >> > > > > with
> > > > > >> > > > > > > big
> > > > > >> > > > > > > > > > table
> > > > > >> > > > > > > > > > > > dbs).
> > > > > >> > > > > > > > > > > > > > > The idea will be to replace generic
> > > > Storage
> > > > > >> > > interface
> > > > > >> > > > > by
> > > > > >> > > > > > > four
> > > > > >> > > > > > > > > > > > specific
> > > > > >> > > > > > > > > > > > > > > interfaces (which will inherit from
> a
> > > > common
> > > > > >> one
> > > > > >> > > that
> > > > > >> > > > > > keeps
> > > > > >> > > > > > > > the
> > > > > >> > > > > > > > > > query
> > > > > >> > > > > > > > > > > > > > part
> > > > > >> > > > > > > > > > > > > > > at is it. with get and query
> methods),
> > > > that
> > > > > >> will
> > > > > >> > > > > include
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > > > required
> > > > > >> > > > > > > > > > > > > > > modification operations for the four
> > > > > DataIndex
> > > > > >> > > > > "domains":
> > > > > >> > > > > > > > > > > > > > processinstance,
> > > > > >> > > > > > > > > > > > > > > usertask, processdefinitions and
> jobs.
> > > > Those
> > > > > >> > > > interfaces
> > > > > >> > > > > > > will
> > > > > >> > > > > > > > > > define
> > > > > >> > > > > > > > > > > > > > methods
> > > > > >> > > > > > > > > > > > > > > like addNode, addVariable,
> updateTask,
> > > > > >> > > > > addAttachment.....
> > > > > >> > > > > > > > that
> > > > > >> > > > > > > > > > will
> > > > > >> > > > > > > > > > > > allow
> > > > > >> > > > > > > > > > > > > > > the persistent layer implementation
> > to
> > > > just
> > > > > >> > update
> > > > > >> > > > the
> > > > > >> > > > > > > > needed
> > > > > >> > > > > > > > > > info
> > > > > >> > > > > > > > > > > > in
> > > > > >> > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > DB  (for example, for addNode in
> > > Postgres,
> > > > > >> just
> > > > > >> > > > insert
> > > > > >> > > > > a
> > > > > >> > > > > > > row
> > > > > >> > > > > > > > > into
> > > > > >> > > > > > > > > > > > nodes
> > > > > >> > > > > > > > > > > > > > > table, for addNode in Mongo,
> basically
> > > the
> > > > > >> same
> > > > > >> > > > atomic
> > > > > >> > > > > > > upsert
> > > > > >> > > > > > > > > > > > operation
> > > > > >> > > > > > > > > > > > > > > that is currently done). Therefore,
> we
> > > > > >> increase
> > > > > >> > > > > > performance
> > > > > >> > > > > > > > for
> > > > > >> > > > > > > > > > > > Postgres
> > > > > >> > > > > > > > > > > > > > > and keep the current one for Mongo.
> > The
> > > > > >> current
> > > > > >> > DB
> > > > > >> > > > > > schemas
> > > > > >> > > > > > > > > won't
> > > > > >> > > > > > > > > > be
> > > > > >> > > > > > > > > > > > > > > touched.
> > > > > >> > > > > > > > > > > > > > > Since the code change is large, I do
> > not
> > > > > think
> > > > > >> > I'll
> > > > > >> > > > be
> > > > > >> > > > > > able
> > > > > >> > > > > > > > to
> > > > > >> > > > > > > > > > have
> > > > > >> > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > PR
> > > > > >> > > > > > > > > > > > > > > ready till next week.
> > > > > >> > > > > > > > > > > > > > > But before starting, please let me
> > know
> > > if
> > > > > >> that
> > > > > >> > > > > approach
> > > > > >> > > > > > is
> > > > > >> > > > > > > > > fine
> > > > > >> > > > > > > > > > for
> > > > > >> > > > > > > > > > > > you.
> > > > > >> > > > > > > > > > > > > > > Best regards.
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 6:55 PM Alex
> > > > > Porcelli
> > > > > >> <
> > > > > >> > > > > > > > > alex@porcelli.me>
> > > > > >> > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > Thank you Francisco to getting
> > deeper
> > > on
> > > > > >> this…
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > Looking forward to see the results
> > of
> > > > your
> > > > > >> > > > suggested
> > > > > >> > > > > > > > > > improvements.
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 9:40 AM
> > > > Francisco
> > > > > >> > Javier
> > > > > >> > > > > Tirado
> > > > > >> > > > > > > > > Sarti <
> > > > > >> > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > I forgot to attach the queries
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 3:04 PM
> > > > > Francisco
> > > > > >> > > Javier
> > > > > >> > > > > > Tirado
> > > > > >> > > > > > > > > > Sarti <
> > > > > >> > > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > >> Hi,
> > > > > >> > > > > > > > > > > > > > > > >> A brief update on this topic.
> > > > > >> > > > > > > > > > > > > > > > >> After doing a simple test with
> > > > example
> > > > > >> > > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > >> > > > > > > > > > > > > > > > ,
> > > > > >> > > > > > > > > > > > > > > > >> the number of updates over
> Nodes
> > > > table
> > > > > is
> > > > > >> > n*n,
> > > > > >> > > > so
> > > > > >> > > > > we
> > > > > >> > > > > > > > > manage
> > > > > >> > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > obtain a
> > > > > >> > > > > > > > > > > > > > > > >> perfect quadratic performance
> > > > > >> degradation.
> > > > > >> > The
> > > > > >> > > > > > problem
> > > > > >> > > > > > > > is
> > > > > >> > > > > > > > > > worse
> > > > > >> > > > > > > > > > > > in
> > > > > >> > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > case
> > > > > >> > > > > > > > > > > > > > > > >> of Serverless Workflow than in
> > BPMN
> > > > > >> because
> > > > > >> > we
> > > > > >> > > > the
> > > > > >> > > > > > > > number
> > > > > >> > > > > > > > > of
> > > > > >> > > > > > > > > > > > nodes
> > > > > >> > > > > > > > > > > > > > is
> > > > > >> > > > > > > > > > > > > > > > >> greater than the number of
> > states.
> > > In
> > > > > >> that
> > > > > >> > > > > example N
> > > > > >> > > > > > > is
> > > > > >> > > > > > > > > 16,
> > > > > >> > > > > > > > > > but
> > > > > >> > > > > > > > > > > > for
> > > > > >> > > > > > > > > > > > > > a
> > > > > >> > > > > > > > > > > > > > > > more
> > > > > >> > > > > > > > > > > > > > > > >> complex workflow it would be
> > > > certainly
> > > > > >> > large.
> > > > > >> > > > > > > > > > > > > > > > >> I think that this is more
> related
> > > to
> > > > > how
> > > > > >> we
> > > > > >> > > are
> > > > > >> > > > > > > handling
> > > > > >> > > > > > > > > > JPA in
> > > > > >> > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > code,
> > > > > >> > > > > > > > > > > > > > > > >> in particular the mapping from
> > > model
> > > > to
> > > > > >> > entity
> > > > > >> > > > > > > > (basically
> > > > > >> > > > > > > > > > JPA is
> > > > > >> > > > > > > > > > > > > > blind
> > > > > >> > > > > > > > > > > > > > > > and
> > > > > >> > > > > > > > > > > > > > > > >> has to update all nodes for
> every
> > > > write
> > > > > >> > > because
> > > > > >> > > > it
> > > > > >> > > > > > > > > believes
> > > > > >> > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > node has
> > > > > >> > > > > > > > > > > > > > > > >> been updated, although it is
> not)
> > > > than
> > > > > an
> > > > > >> > > issue
> > > > > >> > > > in
> > > > > >> > > > > > the
> > > > > >> > > > > > > > > table
> > > > > >> > > > > > > > > > > > > > definition.
> > > > > >> > > > > > > > > > > > > > > > >> In fact, when using JPA,
> > separating
> > > > the
> > > > > >> > server
> > > > > >> > > > > model
> > > > > >> > > > > > > > from
> > > > > >> > > > > > > > > > the
> > > > > >> > > > > > > > > > > > JPA
> > > > > >> > > > > > > > > > > > > > > > entity is
> > > > > >> > > > > > > > > > > > > > > > >> not a good idea, especially if
> > the
> > > > > entity
> > > > > >> > > > contains
> > > > > >> > > > > > > > > > collections.
> > > > > >> > > > > > > > > > > > I
> > > > > >> > > > > > > > > > > > > > will
> > > > > >> > > > > > > > > > > > > > > > try
> > > > > >> > > > > > > > > > > > > > > > >> to change that without breaking
> > > > > anything.
> > > > > >> > > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > > > >> On Wed, Nov 22, 2023 at
> 12:10 PM
> > > > > Enrique
> > > > > >> > > > Gonzalez
> > > > > >> > > > > > > > > Martinez <
> > > > > >> > > > > > > > > > > > > > > > >> egonzalez@apache.org> wrote:
> > > > > >> > > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > > > >>> After the events split you now
> > > will
> > > > > >> need to
> > > > > >> > > > > create
> > > > > >> > > > > > a
> > > > > >> > > > > > > > node
> > > > > >> > > > > > > > > > > > instance
> > > > > >> > > > > > > > > > > > > > > > >>> model instance of making
> > > independent
> > > > > >> from
> > > > > >> > the
> > > > > >> > > > > > process
> > > > > >> > > > > > > > > > instance.
> > > > > >> > > > > > > > > > > > > > > > >>> That should do the trick.
> > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > > > >>> Regarding deleting/inserting
> it
> > > was
> > > > > >> fixed
> > > > > >> > at
> > > > > >> > > > some
> > > > > >> > > > > > > > point.
> > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > > > >>> El mar, 21 nov 2023 a las
> 20:22,
> > > > > >> Francisco
> > > > > >> > > > Javier
> > > > > >> > > > > > > > Tirado
> > > > > >> > > > > > > > > > Sarti
> > > > > >> > > > > > > > > > > > > > > > >>> (<ft...@redhat.com>)
> > escribió:
> > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > >> > > > > > > > > > > > > > > > >>> > Hi Martin,
> > > > > >> > > > > > > > > > > > > > > > >>> > I have a task to review
> > > > performance
> > > > > of
> > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > >> > > > > > > > > > > > > > > > >>> >
> > > ProcessInstanceNodeDataEventMerger
> > > > > >> > > > > > > > > > > > > > > > >>> > My idea is to reduce the
> > number
> > > of
> > > > > >> delete
> > > > > >> > > > > inserts
> > > > > >> > > > > > > > when
> > > > > >> > > > > > > > > > > > processing
> > > > > >> > > > > > > > > > > > > > > > >>> events
> > > > > >> > > > > > > > > > > > > > > > >>> > and try to do it
> incremental.
> > > > > >> > > > > > > > > > > > > > > > >>> > That should improve
> > performance.
> > > > > >> > > > > > > > > > > > > > > > >>> > PS:
> > > > > >> > > > > > > > > > > > > > > > >>> > I was planning to send an
> > e-mail
> > > > > >> tomorrow
> > > > > >> > > > > > > announcing
> > > > > >> > > > > > > > > > that in
> > > > > >> > > > > > > > > > > > > > case you
> > > > > >> > > > > > > > > > > > > > > > >>> were
> > > > > >> > > > > > > > > > > > > > > > >>> > already working on a fix for
> > > > that. I
> > > > > >> > assume
> > > > > >> > > > you
> > > > > >> > > > > > are
> > > > > >> > > > > > > > not
> > > > > >> > > > > > > > > > and I
> > > > > >> > > > > > > > > > > > > > would
> > > > > >> > > > > > > > > > > > > > > > be
> > > > > >> > > > > > > > > > > > > > > > >>> > sending a PR soon.
> > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > >> > > > > > > > > > > > > > > > >>> > On Tue, Nov 21, 2023 at
> > 6:09 PM
> > > > > Martin
> > > > > >> > > Weiler
> > > > > >> > > > > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > >> > > > > > > > > > > > > > > > >>> > wrote:
> > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > >> > > > > > > > > > > > > > > > >>> > > I looked into the new
> > examples
> > > > > using
> > > > > >> > > > > data-index
> > > > > >> > > > > > > > > > persistence
> > > > > >> > > > > > > > > > > > > > addon -
> > > > > >> > > > > > > > > > > > > > > > >>> Neus'
> > > > > >> > > > > > > > > > > > > > > > >>> > > PR#1813 [1] for serverless
> > and
> > > > > >> Pere's
> > > > > >> > > > branch
> > > > > >> > > > > > [2]
> > > > > >> > > > > > > > for
> > > > > >> > > > > > > > > > > > workflow
> > > > > >> > > > > > > > > > > > > > > > (great
> > > > > >> > > > > > > > > > > > > > > > >>> job
> > > > > >> > > > > > > > > > > > > > > > >>> > > both!) - and they work
> > without
> > > > > >> issues
> > > > > >> > > using
> > > > > >> > > > > > > single
> > > > > >> > > > > > > > > > > > requests.
> > > > > >> > > > > > > > > > > > > > > > >>> However, under
> > > > > >> > > > > > > > > > > > > > > > >>> > > some load (I used 'ab' for
> > > > testing
> > > > > >> > with a
> > > > > >> > > > > light
> > > > > >> > > > > > > > > > > > concurrency of
> > > > > >> > > > > > > > > > > > > > 10
> > > > > >> > > > > > > > > > > > > > > > >>> parallel
> > > > > >> > > > > > > > > > > > > > > > >>> > > requests) I ran into the
> > > > following
> > > > > >> > > > problems:
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > > (1) Large number of
> > > > insert/delete
> > > > > >> calls
> > > > > >> > > > (eg.
> > > > > >> > > > > > for
> > > > > >> > > > > > > > > tables
> > > > > >> > > > > > > > > > > > such as
> > > > > >> > > > > > > > > > > > > > > > >>> nodes,
> > > > > >> > > > > > > > > > > > > > > > >>> > > definitions, etc)
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > > (2) Hibernate
> > > > > >> OptimisticLockExceptions
> > > > > >> > /
> > > > > >> > > > > > > > > > > > StaleStateExceptions
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > > (3) DB deadlocks
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > > (4) Error responses, slow
> > > > response
> > > > > >> > times
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > > The reason I am reaching
> out
> > > > with
> > > > > >> this
> > > > > >> > > > topic
> > > > > >> > > > > > here
> > > > > >> > > > > > > > is
> > > > > >> > > > > > > > > to
> > > > > >> > > > > > > > > > > > find
> > > > > >> > > > > > > > > > > > > > out if
> > > > > >> > > > > > > > > > > > > > > > >>> we are
> > > > > >> > > > > > > > > > > > > > > > >>> > > aware of this issue, and
> if
> > > > > someone
> > > > > >> is
> > > > > >> > > > > already
> > > > > >> > > > > > > > > looking
> > > > > >> > > > > > > > > > > > into or
> > > > > >> > > > > > > > > > > > > > > > being
> > > > > >> > > > > > > > > > > > > > > > >>> > > assigned to it?
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > > Thanks,
> > > > > >> > > > > > > > > > > > > > > > >>> > > Martin
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > > [1]
> > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > >
> > > > > >> >
> > https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > >> > > > > > > > > > > > > > > > >>> > > [2]
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > >
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >> > > > > > > > > > > > > > > > >>> > > To unsubscribe, e-mail:
> > > > > >> > > > > > > > > dev-unsubscribe@kie.apache.org
> > > > > >> > > > > > > > > > > > > > > > >>> > > For additional commands,
> > > e-mail:
> > > > > >> > > > > > > > > > dev-help@kie.apache.org
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > >
> > > > > >> >
> > > > ---------------------------------------------------------------------
> > > > > >> > > > > > > > > > > > > > > > >>> To unsubscribe, e-mail:
> > > > > >> > > > > > > dev-unsubscribe@kie.apache.org
> > > > > >> > > > > > > > > > > > > > > > >>> For additional commands,
> e-mail:
> > > > > >> > > > > > > > dev-help@kie.apache.org
> > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > >
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >> > > > > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > > > >> > > > > > dev-unsubscribe@kie.apache.org
> > > > > >> > > > > > > > > > > > > > > > > For additional commands, e-mail:
> > > > > >> > > > > > > dev-help@kie.apache.org
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > >
> > > > > >> >
> > > > ---------------------------------------------------------------------
> > > > > >> > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > > > >> > > dev-unsubscribe@kie.apache.org
> > > > > >> > > > > > > > > > > > > > For additional commands, e-mail:
> > > > > >> > > > dev-help@kie.apache.org
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > >
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >> > > > > > > > > > > > To unsubscribe, e-mail:
> > > > > >> dev-unsubscribe@kie.apache.org
> > > > > >> > > > > > > > > > > > For additional commands, e-mail:
> > > > > >> > dev-help@kie.apache.org
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > >
> > > > > >> >
> > > > ---------------------------------------------------------------------
> > > > > >> > > > > > > > > > To unsubscribe, e-mail:
> > > > > dev-unsubscribe@kie.apache.org
> > > > > >> > > > > > > > > > For additional commands, e-mail:
> > > > > >> dev-help@kie.apache.org
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > >
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >> > > > > > > > > To unsubscribe, e-mail:
> > > > dev-unsubscribe@kie.apache.org
> > > > > >> > > > > > > > > For additional commands, e-mail:
> > > > > dev-help@kie.apache.org
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > >> For additional commands, e-mail: dev-help@kie.apache.org
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Alex Porcelli <al...@porcelli.me>]

Thank you Enrique and Francisco for your input...

I understand the data-index role, it was more a thought provoking statement :)

+1 regarding discussing other stuff first before a comprehensive data
purge approach

On Mon, Feb 19, 2024 at 8:45 AM Francisco Javier Tirado Sarti
<ft...@redhat.com> wrote:
>
> Regarding STP (which is not a concept that we have in the code. I mean STP
> are processes as nonSTP are), I guess, as all processes, they were kept in
> DataIndex once completed because users wanted (and still wants) to check
> the result once the call had been performed. If we want to leave no trace
> of them in DataIndex for some reason, we will need to make it a
> Runtimes concept so DataIndex can handle them in a different way.
>
> On Mon, Feb 19, 2024 at 2:27 PM Enrique Gonzalez Martinez <
> egonzalez@apache.org> wrote:
>
> > Alex:
> > Right now the data index is working in the same way as it did in v7 with
> > the emitters. The only difference between two impl is that in here the
> > storage is pgsql instead elastic search.  You are right regarding is a
> > snapshot of the last state of the process but we did never define how long
> > would be alive that dats Honestly i am happy right now with the way it
> > works. The clean up mechanism is still tbd because we still need to discuss
> > other stuff first.
> >
> >
> > Regarding stp is to leave no trail because u can get the outcome directly
> > from the call. It was defined like that in v7. So there is no use for the
> > index or the audit.
> >
> > El lun, 19 feb 2024, 14:13, Francisco Javier Tirado Sarti <
> > ftirados@redhat.com> escribió:
> >
> > > Hi Alex,
> > > There has been some confusion about the purpose of DataIndex. To be
> > honest
> > > I believe they were already sorted out, but your e-mail makes me think
> > that
> > > is not the case ;). I let Kris to clarify that with you. My view is that
> > > data-index is a way to query recently closed and active processes (the
> > key
> > > here is the definition of recently, which in my opinion should be
> > > configurable)
> > > But, besides that discussion and being pragmatic, keeping finishing
> > process
> > > instances "for a while" in DataIndex was the only way for users to query
> > > the result of straight through processes. That's a function that cannot
> > be
> > > removed right now
> > >
> > > On Mon, Feb 19, 2024 at 1:33 PM Alex Porcelli <po...@apache.org>
> > wrote:
> > >
> > > > if data index was supposed to provide snapshot view of the process
> > > > instance… why do we keep it after the process instance is finished?
> > > >
> > > >
> > > > On Mon, Feb 19, 2024 at 7:12 AM Francisco Javier Tirado Sarti <
> > > > ftirados@redhat.com> wrote:
> > > >
> > > > > Hi Martin.
> > > > > After taking a deeper look at this, I realize that the behaviour is
> > the
> > > > > expected one.
> > > > > Runtimes DB does not track the completed process instance (that's
> > what
> > > > the
> > > > > JDBCProcessInstances warn is telling us), but DataIndex, as expected,
> > > is
> > > > > tracking it in processes and nodes table. And yes it will grow over
> > > time.
> > > > > What we need is some configurable purge mechanism for DataIndex, so
> > it
> > > > > eventually removes older completed process instances.
> > > > >
> > > > > On Tue, Feb 13, 2024 at 12:59 PM Francisco Javier Tirado Sarti <
> > > > > ftirados@redhat.com> wrote:
> > > > >
> > > > > > Hi Martin,
> > > > > > Good catch!. Looks like the skipping performed for process
> > instances
> > > is
> > > > > > not applied to node instances. Something we definitely need to
> > review
> > > > on
> > > > > > the runtimes side.
> > > > > >
> > > > > > On Mon, Feb 12, 2024 at 11:59 PM Martin Weiler
> > > <mweiler@ibm.com.invalid
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> On a somewhat related note, testing a simple workflow (start ->
> > > script
> > > > > >> node -> end), I see the following messages in the logs:
> > > > > >> 2024-02-12 22:49:50,493 28758dde544c WARN
> > > > > >> [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> > > > > >> (executor-thread-3) Skipping create of process instance id:
> > > > > >> 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2
> > > > > >>
> > > > > >> So far, so good. And I'd expect to see no trace of this process in
> > > the
> > > > > >> database if I don't have data audit enabled.
> > > > > >>
> > > > > >> However, the 'processes' table contains a row with state=2, with
> > > > related
> > > > > >> entries in the 'nodes' table. In a load test, I see these tables
> > > grow
> > > > > >> significantly over time. Am I missing something to have these
> > > entries
> > > > > >> cleaned up automatically?
> > > > > >>
> > > > > >> ________________________________________
> > > > > >> From: Martin Weiler <mw...@ibm.com.INVALID>
> > > > > >> Sent: Monday, February 12, 2024 3:40 PM
> > > > > >> To: dev@kie.apache.org
> > > > > >> Subject: [EXTERNAL] RE: [DISCUSSION] Performance issues with
> > > > data-index
> > > > > >> persistence addon
> > > > > >>
> > > > > >> Thanks everyone for your input. Based on this discussion, I opened
> > > the
> > > > > >> following PR:
> > > > > >> https://github.com/apache/incubator-kie-kogito-apps/pull/1985
> > > > > >>
> > > > > >> With this change, the performance seems to be stable over time:
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> > https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
> > > > > >>
> > > > > >> Martin
> > > > > >>
> > > > > >> ________________________________________
> > > > > >> From: Gonzalo Muñoz <gm...@apache.org>
> > > > > >> Sent: Friday, February 9, 2024 9:42 AM
> > > > > >> To: dev@kie.apache.org
> > > > > >> Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > > > data-index
> > > > > >> persistence addon
> > > > > >>
> > > > > >> Great work Francisco,
> > > > > >> Martin, take a look at this link with some related tips (in case
> > you
> > > > > find
> > > > > >> it useful):
> > > > > >> https://www.cybertec-postgresql.com/en/index-your-foreign-key/
> > > > > >>
> > > > > >> El vie, 9 feb 2024 a las 17:20, Francisco Javier Tirado Sarti (<
> > > > > >> ftirados@redhat.com>) escribió:
> > > > > >>
> > > > > >> > For the moment being, we will keep JPA till we exhaust all
> > > > > >> possibilities,
> > > > > >> > let's call switching from jpa to jdbc our hidden plan B ;)
> > > > > >> > I already told Martin, but in order everyone to know, just after
> > > > > writing
> > > > > >> > the previous email, I thought "what if Postgres is not
> > > automatically
> > > > > >> > indexing foreign keys like mysql?" and, eureka
> > > > > >> > Postgres doc
> > > > > >> > https://www.postgresql.org/docs/current/ddl-constraints.html
> > > > > >> > Mysql doc
> > > > > >> >
> > > https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> > > > > >> > These are the relevant excerpt
> > > > > >> >
> > > > > >> > *Postgresql*
> > > > > >> > *A foreign key must reference columns that either are a primary
> > > key
> > > > or
> > > > > >> form
> > > > > >> > a unique constraint, or are columns from a non-partial unique
> > > index.
> > > > > >> This
> > > > > >> > means that the referenced columns always have an index to allow
> > > > > >> efficient
> > > > > >> > lookups on whether a referencing row has a match. Since a DELETE
> > > of
> > > > a
> > > > > >> row
> > > > > >> > from the referenced table or an UPDATE of a referenced column
> > will
> > > > > >> require
> > > > > >> > a scan of the referencing table for rows matching the old value,
> > > it
> > > > is
> > > > > >> > often a good idea to index the referencing columns too. Because
> > > this
> > > > > is
> > > > > >> not
> > > > > >> > always needed, and there are many choices available on how to
> > > index,
> > > > > the
> > > > > >> > declaration of a foreign key constraint does not automatically
> > > > create
> > > > > an
> > > > > >> > index on the referencing columns.*
> > > > > >> > *Mysql*
> > > > > >> > *MySQL requires that foreign key columns be indexed; if you
> > > create a
> > > > > >> table
> > > > > >> > with a foreign key constraint but no index on a given column, an
> > > > index
> > > > > >> is
> > > > > >> > created. *
> > > > > >> >
> > > > > >> > So I asked Martin to especially create an index for
> > > > > process_instance_id
> > > > > >> > column on nodes table
> > > > > >> > I think that will fix the problem detected on the thread dump.
> > > > > >> > The simpler process test to verify queries are fine still
> > stands,
> > > > > >> though ;)
> > > > > >> >
> > > > > >> >
> > > > > >> > On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <
> > tzimanyi@apache.org
> > > >
> > > > > >> wrote:
> > > > > >> >
> > > > > >> > > I always preferred pure JDBC over Hibernate myself, just for
> > the
> > > > > sake
> > > > > >> of
> > > > > >> > > control of what is happening :) So I would not -1 that myself.
> > > > > >> > >
> > > > > >> > > Tibor
> > > > > >> > >
> > > > > >> > > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado Sarti <
> > > > > >> > > ftirados@redhat.com>
> > > > > >> > > napísal(a):
> > > > > >> > >
> > > > > >> > > > Hi,
> > > > > >> > > > Usually I do not want to talk about work in progress because
> > > > > >> > preliminary
> > > > > >> > > > conclusions are pretty volatile but, well, there are a
> > couple
> > > of
> > > > > >> things
> > > > > >> > > > that can be concluded from the really valuable information
> > > that
> > > > > >> Martin
> > > > > >> > > > provided.
> > > > > >> > > > 1) In order to be able to determine if the number of
> > > statements
> > > > is
> > > > > >> > larger
> > > > > >> > > > than expected, I asked Martin to test with a simpler process
> > > > > >> > definition.
> > > > > >> > > > One with just three nodes: start, script and end. The script
> > > one
> > > > > >> should
> > > > > >> > > > change just one variable. This way we can analyze if the
> > > number
> > > > of
> > > > > >> > > queries
> > > > > >> > > > is the expected one. From the single log (audit was
> > activated
> > > > > them)
> > > > > >> my
> > > > > >> > > > conclusion is that the number of insert/updates over
> > processes
> > > > and
> > > > > >> > nodes
> > > > > >> > > > (there a lot over task, that I will prefer to skip for now,
> > > baby
> > > > > >> steps)
> > > > > >> > > is
> > > > > >> > > > the expected one.
> > > > > >> > > > 2) Analysing the thread dump, we see around 15 threads
> > > executing
> > > > > >> this
> > > > > >> > > line
> > > > > >> > > > at
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> > org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > > > >> > > > so its pretty clear the code to be optimized ;). I'm
> > > evaluating
> > > > > >> > > > possibilities within JPA/Hibernate, but I'm starting to
> > think
> > > > that
> > > > > >> it
> > > > > >> > > might
> > > > > >> > > > be better to switch to JDBC and skip hibernate. Our lives
> > will
> > > > be
> > > > > >> > > simpler,
> > > > > >> > > > especially with a schema relatively simple like ours (that
> > > will
> > > > be
> > > > > >> my
> > > > > >> > > > recommendation if I was an external consultant)
> > > > > >> > > >
> > > > > >> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <
> > > > tzimanyi@apache.org
> > > > > >
> > > > > >> > > wrote:
> > > > > >> > > >
> > > > > >> > > > > Hi,
> > > > > >> > > > >
> > > > > >> > > > > this will be a bit off-topic. However as far as
> > > performance, I
> > > > > >> think
> > > > > >> > we
> > > > > >> > > > > should think about that we have string primary keys
> > (IDs). I
> > > > > would
> > > > > >> > > expect
> > > > > >> > > > > the database systems are much better with indexing numeric
> > > > keys
> > > > > >> than
> > > > > >> > > > > strings. I remember from the past, when I was working with
> > > > DBs,
> > > > > >> that
> > > > > >> > > > using
> > > > > >> > > > > strings as keys or indexes was a discouraged practice.
> > > > > >> > > > >
> > > > > >> > > > > Best regards,
> > > > > >> > > > > Tibor
> > > > > >> > > > >
> > > > > >> > > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler
> > > > <mweiler@ibm.com.invalid
> > > > > >
> > > > > >> > > > > napísal(a):
> > > > > >> > > > >
> > > > > >> > > > > > I changed the test to use MongoDB [1] and I don't see a
> > > > > >> performance
> > > > > >> > > > > > degradation with this setup [2].
> > > > > >> > > > > >
> > > > > >> > > > > > Please keep us posted of your findings. Thanks!
> > > > > >> > > > > >
> > > > > >> > > > > > Martin
> > > > > >> > > > > >
> > > > > >> > > > > > [1]
> > > > > >> > > > >
> > > > > >> >
> > > > >
> > https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > > >> > > > > > [2]
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> > https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > > >> > > > > >
> > > > > >> > > > > > ________________________________________
> > > > > >> > > > > > From: Francisco Javier Tirado Sarti <
> > ftirados@redhat.com>
> > > > > >> > > > > > Sent: Wednesday, February 7, 2024 11:40 AM
> > > > > >> > > > > > To: dev@kie.apache.org
> > > > > >> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues
> > > with
> > > > > >> > > data-index
> > > > > >> > > > > > persistence addon
> > > > > >> > > > > >
> > > > > >> > > > > > yes, it can be index degradation because of size, but I
> > > > > believe
> > > > > >> (I
> > > > > >> > > > might
> > > > > >> > > > > be
> > > > > >> > > > > > wrong) the db is too small (yet) for that.
> > > > > >> > > > > > But, eventually, Postgres, when the DB is huge enough,
> > > > > >> unavoidably
> > > > > >> > > will
> > > > > >> > > > > > behave like the graphic that Martin sent.
> > > > > >> > > > > > Since I believe we are not huge enough (yet), lets rule
> > > out
> > > > > >> another
> > > > > >> > > > issue
> > > > > >> > > > > > by analysing the sql logs (I requested those to Martin
> > > > offline
> > > > > >> and
> > > > > >> > he
> > > > > >> > > > is
> > > > > >> > > > > > going to kindly collect them).
> > > > > >> > > > > > Also Im curious to know if Mongo behave in the same way.
> > > > > >> > > > > >
> > > > > >> > > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez
> > Martinez <
> > > > > >> > > > > > egonzalez@apache.org> wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > > Hi Francisco,
> > > > > >> > > > > > > I would highly recommend to check indexes and how the
> > > > > updates
> > > > > >> > work
> > > > > >> > > in
> > > > > >> > > > > > data
> > > > > >> > > > > > > index to avoid full scan table and lock the full
> > table.
> > > > Some
> > > > > >> db
> > > > > >> > are
> > > > > >> > > > > very
> > > > > >> > > > > > > sensitive to that.
> > > > > >> > > > > > >
> > > > > >> > > > > > > El mié, 7 feb 2024, 18:41, Francisco Javier Tirado
> > > Sarti <
> > > > > >> > > > > > > ftirados@redhat.com> escribió:
> > > > > >> > > > > > >
> > > > > >> > > > > > > > Hi Martin,
> > > > > >> > > > > > > > While I analyze the data, let me ask you if it is
> > > > possible
> > > > > >> to
> > > > > >> > > > perform
> > > > > >> > > > > > > > another check (similar in a way to disabling
> > > data-index
> > > > > like
> > > > > >> > you
> > > > > >> > > > do)
> > > > > >> > > > > > Can
> > > > > >> > > > > > > > you switch to MongoDB persistence and check if the
> > > same
> > > > > >> > > degradation
> > > > > >> > > > > > that
> > > > > >> > > > > > > is
> > > > > >> > > > > > > > there for postgres remains?
> > > > > >> > > > > > > > I do not know if this is feasible but will certainly
> > > > > >> indicate
> > > > > >> > the
> > > > > >> > > > > > problem
> > > > > >> > > > > > > > is on the postgres storage layer and I do not have a
> > > > clear
> > > > > >> > > > prediction
> > > > > >> > > > > > of
> > > > > >> > > > > > > > what we will see when doing this switch.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler
> > > > > >> > > > <mweiler@ibm.com.invalid
> > > > > >> > > > > >
> > > > > >> > > > > > > > wrote:
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > Hi Francisco,
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > thanks for your work on this important topic!
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > I would like to share some test results here,
> > which
> > > > > might
> > > > > >> > help
> > > > > >> > > to
> > > > > >> > > > > > > improve
> > > > > >> > > > > > > > > the codebase even further. I am using the jmeter
> > > based
> > > > > >> test
> > > > > >> > > case
> > > > > >> > > > > from
> > > > > >> > > > > > > > Pere
> > > > > >> > > > > > > > > and Enrique (thanks guys!) [1] which uses a load
> > of
> > > 30
> > > > > >> > threads
> > > > > >> > > to
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > 1) start a new process instance (POST)
> > > > > >> > > > > > > > > 2) retrieve tasks for a user (GET)
> > > > > >> > > > > > > > > 3) fetches task details (GET)
> > > > > >> > > > > > > > > 4) complete a task (POST)
> > > > > >> > > > > > > > > 5) execute a query on data-audit
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > With this test setup, I noticed that the
> > performance
> > > > for
> > > > > >> the
> > > > > >> > > POST
> > > > > >> > > > > > > > > requests, in particular the one to start a new
> > > process
> > > > > >> > > instance,
> > > > > >> > > > > > > degrades
> > > > > >> > > > > > > > > over time - see graph [2]. If I run the same test
> > > > > without
> > > > > >> > > > > data-index,
> > > > > >> > > > > > > > then
> > > > > >> > > > > > > > > there is no such performance degradation [3]. You
> > > can
> > > > > >> find a
> > > > > >> > > > thread
> > > > > >> > > > > > > dump
> > > > > >> > > > > > > > > captured a few minutes into the first test here
> > [4]
> > > > that
> > > > > >> > might
> > > > > >> > > > help
> > > > > >> > > > > > to
> > > > > >> > > > > > > > see
> > > > > >> > > > > > > > > some of the contention points.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > I'd appreciate if you could take a look and see if
> > > > there
> > > > > >> is
> > > > > >> > > > > something
> > > > > >> > > > > > > > that
> > > > > >> > > > > > > > > can be further improved based on your previous
> > work.
> > > > If
> > > > > >> you
> > > > > >> > > need
> > > > > >> > > > > any
> > > > > >> > > > > > > > > additional data, let me know, but otherwise it is
> > > > > >> > > straightforward
> > > > > >> > > > > to
> > > > > >> > > > > > > run
> > > > > >> > > > > > > > > the jmeter test as well.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Thanks,
> > > > > >> > > > > > > > > Martin
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > [1]
> > > > > >> https://github.com/pefernan/job-service-refactor-test/
> > > > > >> > > > > > > > > [2]
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> > https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > >> > > > > > > > > [3]
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> > https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > >> > > > > > > > > [4]
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> > https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > ________________________________________
> > > > > >> > > > > > > > > From: Francisco Javier Tirado Sarti <
> > > > > ftirados@redhat.com>
> > > > > >> > > > > > > > > Sent: Wednesday, January 17, 2024 9:13 AM
> > > > > >> > > > > > > > > To: dev@kie.apache.org
> > > > > >> > > > > > > > > Cc: Pere Fernandez Perez
> > > > > >> > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance
> > > > issues
> > > > > >> with
> > > > > >> > > > > > data-index
> > > > > >> > > > > > > > > persistence addon
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Hi Alex,
> > > > > >> > > > > > > > > I did not take times (which depends on a number of
> > > > > >> variables
> > > > > >> > > that
> > > > > >> > > > > > > > > drastically change between environments), but
> > verify
> > > > > that
> > > > > >> the
> > > > > >> > > > > number
> > > > > >> > > > > > of
> > > > > >> > > > > > > > > updates has been reduced drastically without
> > losing
> > > > > >> > > > functionality,
> > > > > >> > > > > > > which
> > > > > >> > > > > > > > is
> > > > > >> > > > > > > > > objectively a good thing. If before the change,
> > for
> > > > > every
> > > > > >> > node
> > > > > >> > > > > > > executed,
> > > > > >> > > > > > > > we
> > > > > >> > > > > > > > > have an update for every node previously executed,
> > > so
> > > > > if a
> > > > > >> > > > process
> > > > > >> > > > > > have
> > > > > >> > > > > > > > 50
> > > > > >> > > > > > > > > nodes to execute, we were performing nearly
> > 50*51/2
> > > > > >> updates,
> > > > > >> > > > which
> > > > > >> > > > > > > gives
> > > > > >> > > > > > > > us
> > > > > >> > > > > > > > > a total of  1275 updates, now we have just one for
> > > > every
> > > > > >> node
> > > > > >> > > > being
> > > > > >> > > > > > > > > executed, implying a total of 50 updates.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli <
> > > > > >> > > alex@porcelli.me>
> > > > > >> > > > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > Francisco,
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > I noticed that your PR has been merged, but I
> > was
> > > > > >> expecting
> > > > > >> > > (at
> > > > > >> > > > > > least
> > > > > >> > > > > > > > > > was my understanding from this thread) that
> > before
> > > > > >> merging
> > > > > >> > > some
> > > > > >> > > > > > > > > > benchmark data would be shared in advance - to
> > > > assess
> > > > > >> the
> > > > > >> > > > > > > cost/benefit
> > > > > >> > > > > > > > > > of such a decent size change.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Do you have any information to share?
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM Francisco Javier
> > > > > Tirado
> > > > > >> > Sarti
> > > > > >> > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Yes, as intended, now we have one select and
> > one
> > > > > >> > > > insert/update
> > > > > >> > > > > > per
> > > > > >> > > > > > > > node
> > > > > >> > > > > > > > > > > event.
> > > > > >> > > > > > > > > > > I moved the PR as ready for review and give
> > > @Pere
> > > > > >> > Fernandez
> > > > > >> > > > > Perez
> > > > > >> > > > > > > > > > > <pe...@redhat.com> permission to the
> > branch
> > > so
> > > > > he
> > > > > >> can
> > > > > >> > > > edit
> > > > > >> > > > > it
> > > > > >> > > > > > > in
> > > > > >> > > > > > > > > the
> > > > > >> > > > > > > > > > > next two weeks (Ill be on PTO)  if desired,
> > > before
> > > > > >> > merging.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex Porcelli
> > <
> > > > > >> > > > > alex@porcelli.me>
> > > > > >> > > > > > > > > wrote:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Cool, thank you Francisco!
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Did you manage to get some preliminary data
> > > > about
> > > > > >> > > > > improvements?
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM Francisco
> > > > Javier
> > > > > >> > Tirado
> > > > > >> > > > > Sarti
> > > > > >> > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Yes, after some delay because of quarkus 3
> > > > > >> migration.
> > > > > >> > > Im
> > > > > >> > > > > > > refining
> > > > > >> > > > > > > > > > this
> > > > > >> > > > > > > > > > > > > draft PR
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > >
> > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex
> > > Porcelli
> > > > <
> > > > > >> > > > > > > alex@porcelli.me>
> > > > > >> > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Any update or new findings on this
> > topic?
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM
> > Francisco
> > > > > Javier
> > > > > >> > > Tirado
> > > > > >> > > > > > Sarti
> > > > > >> > > > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > Hi Alex,
> > > > > >> > > > > > > > > > > > > > > After considering different options to
> > > > > improve
> > > > > >> > > > > > performance,
> > > > > >> > > > > > > > we
> > > > > >> > > > > > > > > > feel
> > > > > >> > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > > it
> > > > > >> > > > > > > > > > > > > > > is time to "partially" move away from
> > > the
> > > > > >> current
> > > > > >> > > Map
> > > > > >> > > > > > style
> > > > > >> > > > > > > > > > > > interface (
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> > https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > >> > > > > > > > > > > > > > )
> > > > > >> > > > > > > > > > > > > > > which was shared with Trusty, to one
> > > more
> > > > > >> > suitable
> > > > > >> > > > for
> > > > > >> > > > > > > usage
> > > > > >> > > > > > > > > > with a
> > > > > >> > > > > > > > > > > > > > > relational DB like postgresql (but
> > still
> > > > > >> > compatible
> > > > > >> > > > > with
> > > > > >> > > > > > > big
> > > > > >> > > > > > > > > > table
> > > > > >> > > > > > > > > > > > dbs).
> > > > > >> > > > > > > > > > > > > > > The idea will be to replace generic
> > > > Storage
> > > > > >> > > interface
> > > > > >> > > > > by
> > > > > >> > > > > > > four
> > > > > >> > > > > > > > > > > > specific
> > > > > >> > > > > > > > > > > > > > > interfaces (which will inherit from a
> > > > common
> > > > > >> one
> > > > > >> > > that
> > > > > >> > > > > > keeps
> > > > > >> > > > > > > > the
> > > > > >> > > > > > > > > > query
> > > > > >> > > > > > > > > > > > > > part
> > > > > >> > > > > > > > > > > > > > > at is it. with get and query methods),
> > > > that
> > > > > >> will
> > > > > >> > > > > include
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > > > required
> > > > > >> > > > > > > > > > > > > > > modification operations for the four
> > > > > DataIndex
> > > > > >> > > > > "domains":
> > > > > >> > > > > > > > > > > > > > processinstance,
> > > > > >> > > > > > > > > > > > > > > usertask, processdefinitions and jobs.
> > > > Those
> > > > > >> > > > interfaces
> > > > > >> > > > > > > will
> > > > > >> > > > > > > > > > define
> > > > > >> > > > > > > > > > > > > > methods
> > > > > >> > > > > > > > > > > > > > > like addNode, addVariable, updateTask,
> > > > > >> > > > > addAttachment.....
> > > > > >> > > > > > > > that
> > > > > >> > > > > > > > > > will
> > > > > >> > > > > > > > > > > > allow
> > > > > >> > > > > > > > > > > > > > > the persistent layer implementation
> > to
> > > > just
> > > > > >> > update
> > > > > >> > > > the
> > > > > >> > > > > > > > needed
> > > > > >> > > > > > > > > > info
> > > > > >> > > > > > > > > > > > in
> > > > > >> > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > DB  (for example, for addNode in
> > > Postgres,
> > > > > >> just
> > > > > >> > > > insert
> > > > > >> > > > > a
> > > > > >> > > > > > > row
> > > > > >> > > > > > > > > into
> > > > > >> > > > > > > > > > > > nodes
> > > > > >> > > > > > > > > > > > > > > table, for addNode in Mongo, basically
> > > the
> > > > > >> same
> > > > > >> > > > atomic
> > > > > >> > > > > > > upsert
> > > > > >> > > > > > > > > > > > operation
> > > > > >> > > > > > > > > > > > > > > that is currently done). Therefore, we
> > > > > >> increase
> > > > > >> > > > > > performance
> > > > > >> > > > > > > > for
> > > > > >> > > > > > > > > > > > Postgres
> > > > > >> > > > > > > > > > > > > > > and keep the current one for Mongo.
> > The
> > > > > >> current
> > > > > >> > DB
> > > > > >> > > > > > schemas
> > > > > >> > > > > > > > > won't
> > > > > >> > > > > > > > > > be
> > > > > >> > > > > > > > > > > > > > > touched.
> > > > > >> > > > > > > > > > > > > > > Since the code change is large, I do
> > not
> > > > > think
> > > > > >> > I'll
> > > > > >> > > > be
> > > > > >> > > > > > able
> > > > > >> > > > > > > > to
> > > > > >> > > > > > > > > > have
> > > > > >> > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > PR
> > > > > >> > > > > > > > > > > > > > > ready till next week.
> > > > > >> > > > > > > > > > > > > > > But before starting, please let me
> > know
> > > if
> > > > > >> that
> > > > > >> > > > > approach
> > > > > >> > > > > > is
> > > > > >> > > > > > > > > fine
> > > > > >> > > > > > > > > > for
> > > > > >> > > > > > > > > > > > you.
> > > > > >> > > > > > > > > > > > > > > Best regards.
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 6:55 PM Alex
> > > > > Porcelli
> > > > > >> <
> > > > > >> > > > > > > > > alex@porcelli.me>
> > > > > >> > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > Thank you Francisco to getting
> > deeper
> > > on
> > > > > >> this…
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > Looking forward to see the results
> > of
> > > > your
> > > > > >> > > > suggested
> > > > > >> > > > > > > > > > improvements.
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 9:40 AM
> > > > Francisco
> > > > > >> > Javier
> > > > > >> > > > > Tirado
> > > > > >> > > > > > > > > Sarti <
> > > > > >> > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > I forgot to attach the queries
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 3:04 PM
> > > > > Francisco
> > > > > >> > > Javier
> > > > > >> > > > > > Tirado
> > > > > >> > > > > > > > > > Sarti <
> > > > > >> > > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > >> Hi,
> > > > > >> > > > > > > > > > > > > > > > >> A brief update on this topic.
> > > > > >> > > > > > > > > > > > > > > > >> After doing a simple test with
> > > > example
> > > > > >> > > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> > https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > >> > > > > > > > > > > > > > > > ,
> > > > > >> > > > > > > > > > > > > > > > >> the number of updates over Nodes
> > > > table
> > > > > is
> > > > > >> > n*n,
> > > > > >> > > > so
> > > > > >> > > > > we
> > > > > >> > > > > > > > > manage
> > > > > >> > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > obtain a
> > > > > >> > > > > > > > > > > > > > > > >> perfect quadratic performance
> > > > > >> degradation.
> > > > > >> > The
> > > > > >> > > > > > problem
> > > > > >> > > > > > > > is
> > > > > >> > > > > > > > > > worse
> > > > > >> > > > > > > > > > > > in
> > > > > >> > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > case
> > > > > >> > > > > > > > > > > > > > > > >> of Serverless Workflow than in
> > BPMN
> > > > > >> because
> > > > > >> > we
> > > > > >> > > > the
> > > > > >> > > > > > > > number
> > > > > >> > > > > > > > > of
> > > > > >> > > > > > > > > > > > nodes
> > > > > >> > > > > > > > > > > > > > is
> > > > > >> > > > > > > > > > > > > > > > >> greater than the number of
> > states.
> > > In
> > > > > >> that
> > > > > >> > > > > example N
> > > > > >> > > > > > > is
> > > > > >> > > > > > > > > 16,
> > > > > >> > > > > > > > > > but
> > > > > >> > > > > > > > > > > > for
> > > > > >> > > > > > > > > > > > > > a
> > > > > >> > > > > > > > > > > > > > > > more
> > > > > >> > > > > > > > > > > > > > > > >> complex workflow it would be
> > > > certainly
> > > > > >> > large.
> > > > > >> > > > > > > > > > > > > > > > >> I think that this is more related
> > > to
> > > > > how
> > > > > >> we
> > > > > >> > > are
> > > > > >> > > > > > > handling
> > > > > >> > > > > > > > > > JPA in
> > > > > >> > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > > code,
> > > > > >> > > > > > > > > > > > > > > > >> in particular the mapping from
> > > model
> > > > to
> > > > > >> > entity
> > > > > >> > > > > > > > (basically
> > > > > >> > > > > > > > > > JPA is
> > > > > >> > > > > > > > > > > > > > blind
> > > > > >> > > > > > > > > > > > > > > > and
> > > > > >> > > > > > > > > > > > > > > > >> has to update all nodes for every
> > > > write
> > > > > >> > > because
> > > > > >> > > > it
> > > > > >> > > > > > > > > believes
> > > > > >> > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > node has
> > > > > >> > > > > > > > > > > > > > > > >> been updated, although it is not)
> > > > than
> > > > > an
> > > > > >> > > issue
> > > > > >> > > > in
> > > > > >> > > > > > the
> > > > > >> > > > > > > > > table
> > > > > >> > > > > > > > > > > > > > definition.
> > > > > >> > > > > > > > > > > > > > > > >> In fact, when using JPA,
> > separating
> > > > the
> > > > > >> > server
> > > > > >> > > > > model
> > > > > >> > > > > > > > from
> > > > > >> > > > > > > > > > the
> > > > > >> > > > > > > > > > > > JPA
> > > > > >> > > > > > > > > > > > > > > > entity is
> > > > > >> > > > > > > > > > > > > > > > >> not a good idea, especially if
> > the
> > > > > entity
> > > > > >> > > > contains
> > > > > >> > > > > > > > > > collections.
> > > > > >> > > > > > > > > > > > I
> > > > > >> > > > > > > > > > > > > > will
> > > > > >> > > > > > > > > > > > > > > > try
> > > > > >> > > > > > > > > > > > > > > > >> to change that without breaking
> > > > > anything.
> > > > > >> > > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > > > >> On Wed, Nov 22, 2023 at 12:10 PM
> > > > > Enrique
> > > > > >> > > > Gonzalez
> > > > > >> > > > > > > > > Martinez <
> > > > > >> > > > > > > > > > > > > > > > >> egonzalez@apache.org> wrote:
> > > > > >> > > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > > > >>> After the events split you now
> > > will
> > > > > >> need to
> > > > > >> > > > > create
> > > > > >> > > > > > a
> > > > > >> > > > > > > > node
> > > > > >> > > > > > > > > > > > instance
> > > > > >> > > > > > > > > > > > > > > > >>> model instance of making
> > > independent
> > > > > >> from
> > > > > >> > the
> > > > > >> > > > > > process
> > > > > >> > > > > > > > > > instance.
> > > > > >> > > > > > > > > > > > > > > > >>> That should do the trick.
> > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > > > >>> Regarding deleting/inserting it
> > > was
> > > > > >> fixed
> > > > > >> > at
> > > > > >> > > > some
> > > > > >> > > > > > > > point.
> > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > > > >>> El mar, 21 nov 2023 a las 20:22,
> > > > > >> Francisco
> > > > > >> > > > Javier
> > > > > >> > > > > > > > Tirado
> > > > > >> > > > > > > > > > Sarti
> > > > > >> > > > > > > > > > > > > > > > >>> (<ft...@redhat.com>)
> > escribió:
> > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > >> > > > > > > > > > > > > > > > >>> > Hi Martin,
> > > > > >> > > > > > > > > > > > > > > > >>> > I have a task to review
> > > > performance
> > > > > of
> > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > >> > > > > > > > > > > > > > > > >>> >
> > > ProcessInstanceNodeDataEventMerger
> > > > > >> > > > > > > > > > > > > > > > >>> > My idea is to reduce the
> > number
> > > of
> > > > > >> delete
> > > > > >> > > > > inserts
> > > > > >> > > > > > > > when
> > > > > >> > > > > > > > > > > > processing
> > > > > >> > > > > > > > > > > > > > > > >>> events
> > > > > >> > > > > > > > > > > > > > > > >>> > and try to do it incremental.
> > > > > >> > > > > > > > > > > > > > > > >>> > That should improve
> > performance.
> > > > > >> > > > > > > > > > > > > > > > >>> > PS:
> > > > > >> > > > > > > > > > > > > > > > >>> > I was planning to send an
> > e-mail
> > > > > >> tomorrow
> > > > > >> > > > > > > announcing
> > > > > >> > > > > > > > > > that in
> > > > > >> > > > > > > > > > > > > > case you
> > > > > >> > > > > > > > > > > > > > > > >>> were
> > > > > >> > > > > > > > > > > > > > > > >>> > already working on a fix for
> > > > that. I
> > > > > >> > assume
> > > > > >> > > > you
> > > > > >> > > > > > are
> > > > > >> > > > > > > > not
> > > > > >> > > > > > > > > > and I
> > > > > >> > > > > > > > > > > > > > would
> > > > > >> > > > > > > > > > > > > > > > be
> > > > > >> > > > > > > > > > > > > > > > >>> > sending a PR soon.
> > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > >> > > > > > > > > > > > > > > > >>> > On Tue, Nov 21, 2023 at
> > 6:09 PM
> > > > > Martin
> > > > > >> > > Weiler
> > > > > >> > > > > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > >> > > > > > > > > > > > > > > > >>> > wrote:
> > > > > >> > > > > > > > > > > > > > > > >>> >
> > > > > >> > > > > > > > > > > > > > > > >>> > > I looked into the new
> > examples
> > > > > using
> > > > > >> > > > > data-index
> > > > > >> > > > > > > > > > persistence
> > > > > >> > > > > > > > > > > > > > addon -
> > > > > >> > > > > > > > > > > > > > > > >>> Neus'
> > > > > >> > > > > > > > > > > > > > > > >>> > > PR#1813 [1] for serverless
> > and
> > > > > >> Pere's
> > > > > >> > > > branch
> > > > > >> > > > > > [2]
> > > > > >> > > > > > > > for
> > > > > >> > > > > > > > > > > > workflow
> > > > > >> > > > > > > > > > > > > > > > (great
> > > > > >> > > > > > > > > > > > > > > > >>> job
> > > > > >> > > > > > > > > > > > > > > > >>> > > both!) - and they work
> > without
> > > > > >> issues
> > > > > >> > > using
> > > > > >> > > > > > > single
> > > > > >> > > > > > > > > > > > requests.
> > > > > >> > > > > > > > > > > > > > > > >>> However, under
> > > > > >> > > > > > > > > > > > > > > > >>> > > some load (I used 'ab' for
> > > > testing
> > > > > >> > with a
> > > > > >> > > > > light
> > > > > >> > > > > > > > > > > > concurrency of
> > > > > >> > > > > > > > > > > > > > 10
> > > > > >> > > > > > > > > > > > > > > > >>> parallel
> > > > > >> > > > > > > > > > > > > > > > >>> > > requests) I ran into the
> > > > following
> > > > > >> > > > problems:
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > > (1) Large number of
> > > > insert/delete
> > > > > >> calls
> > > > > >> > > > (eg.
> > > > > >> > > > > > for
> > > > > >> > > > > > > > > tables
> > > > > >> > > > > > > > > > > > such as
> > > > > >> > > > > > > > > > > > > > > > >>> nodes,
> > > > > >> > > > > > > > > > > > > > > > >>> > > definitions, etc)
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > > (2) Hibernate
> > > > > >> OptimisticLockExceptions
> > > > > >> > /
> > > > > >> > > > > > > > > > > > StaleStateExceptions
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > > (3) DB deadlocks
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > > (4) Error responses, slow
> > > > response
> > > > > >> > times
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > > The reason I am reaching out
> > > > with
> > > > > >> this
> > > > > >> > > > topic
> > > > > >> > > > > > here
> > > > > >> > > > > > > > is
> > > > > >> > > > > > > > > to
> > > > > >> > > > > > > > > > > > find
> > > > > >> > > > > > > > > > > > > > out if
> > > > > >> > > > > > > > > > > > > > > > >>> we are
> > > > > >> > > > > > > > > > > > > > > > >>> > > aware of this issue, and if
> > > > > someone
> > > > > >> is
> > > > > >> > > > > already
> > > > > >> > > > > > > > > looking
> > > > > >> > > > > > > > > > > > into or
> > > > > >> > > > > > > > > > > > > > > > being
> > > > > >> > > > > > > > > > > > > > > > >>> > > assigned to it?
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > > Thanks,
> > > > > >> > > > > > > > > > > > > > > > >>> > > Martin
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > > [1]
> > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > >
> > > > > >> >
> > https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > >> > > > > > > > > > > > > > > > >>> > > [2]
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> > https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > >
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >> > > > > > > > > > > > > > > > >>> > > To unsubscribe, e-mail:
> > > > > >> > > > > > > > > dev-unsubscribe@kie.apache.org
> > > > > >> > > > > > > > > > > > > > > > >>> > > For additional commands,
> > > e-mail:
> > > > > >> > > > > > > > > > dev-help@kie.apache.org
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > >
> > > > > >> >
> > > > ---------------------------------------------------------------------
> > > > > >> > > > > > > > > > > > > > > > >>> To unsubscribe, e-mail:
> > > > > >> > > > > > > dev-unsubscribe@kie.apache.org
> > > > > >> > > > > > > > > > > > > > > > >>> For additional commands, e-mail:
> > > > > >> > > > > > > > dev-help@kie.apache.org
> > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > >
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >> > > > > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > > > >> > > > > > dev-unsubscribe@kie.apache.org
> > > > > >> > > > > > > > > > > > > > > > > For additional commands, e-mail:
> > > > > >> > > > > > > dev-help@kie.apache.org
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > >
> > > > > >> >
> > > > ---------------------------------------------------------------------
> > > > > >> > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > > > >> > > dev-unsubscribe@kie.apache.org
> > > > > >> > > > > > > > > > > > > > For additional commands, e-mail:
> > > > > >> > > > dev-help@kie.apache.org
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > >
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >> > > > > > > > > > > > To unsubscribe, e-mail:
> > > > > >> dev-unsubscribe@kie.apache.org
> > > > > >> > > > > > > > > > > > For additional commands, e-mail:
> > > > > >> > dev-help@kie.apache.org
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > >
> > > > > >> >
> > > > ---------------------------------------------------------------------
> > > > > >> > > > > > > > > > To unsubscribe, e-mail:
> > > > > dev-unsubscribe@kie.apache.org
> > > > > >> > > > > > > > > > For additional commands, e-mail:
> > > > > >> dev-help@kie.apache.org
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > >
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >> > > > > > > > > To unsubscribe, e-mail:
> > > > dev-unsubscribe@kie.apache.org
> > > > > >> > > > > > > > > For additional commands, e-mail:
> > > > > dev-help@kie.apache.org
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > >> For additional commands, e-mail: dev-help@kie.apache.org
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
For additional commands, e-mail: dev-help@kie.apache.org

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Francisco Javier Tirado Sarti <ft...@redhat.com>]

Regarding STP (which is not a concept that we have in the code. I mean STP
are processes as nonSTP are), I guess, as all processes, they were kept in
DataIndex once completed because users wanted (and still wants) to check
the result once the call had been performed. If we want to leave no trace
of them in DataIndex for some reason, we will need to make it a
Runtimes concept so DataIndex can handle them in a different way.

On Mon, Feb 19, 2024 at 2:27 PM Enrique Gonzalez Martinez <
egonzalez@apache.org> wrote:

> Alex:
> Right now the data index is working in the same way as it did in v7 with
> the emitters. The only difference between two impl is that in here the
> storage is pgsql instead elastic search.  You are right regarding is a
> snapshot of the last state of the process but we did never define how long
> would be alive that dats Honestly i am happy right now with the way it
> works. The clean up mechanism is still tbd because we still need to discuss
> other stuff first.
>
>
> Regarding stp is to leave no trail because u can get the outcome directly
> from the call. It was defined like that in v7. So there is no use for the
> index or the audit.
>
> El lun, 19 feb 2024, 14:13, Francisco Javier Tirado Sarti <
> ftirados@redhat.com> escribió:
>
> > Hi Alex,
> > There has been some confusion about the purpose of DataIndex. To be
> honest
> > I believe they were already sorted out, but your e-mail makes me think
> that
> > is not the case ;). I let Kris to clarify that with you. My view is that
> > data-index is a way to query recently closed and active processes (the
> key
> > here is the definition of recently, which in my opinion should be
> > configurable)
> > But, besides that discussion and being pragmatic, keeping finishing
> process
> > instances "for a while" in DataIndex was the only way for users to query
> > the result of straight through processes. That's a function that cannot
> be
> > removed right now
> >
> > On Mon, Feb 19, 2024 at 1:33 PM Alex Porcelli <po...@apache.org>
> wrote:
> >
> > > if data index was supposed to provide snapshot view of the process
> > > instance… why do we keep it after the process instance is finished?
> > >
> > >
> > > On Mon, Feb 19, 2024 at 7:12 AM Francisco Javier Tirado Sarti <
> > > ftirados@redhat.com> wrote:
> > >
> > > > Hi Martin.
> > > > After taking a deeper look at this, I realize that the behaviour is
> the
> > > > expected one.
> > > > Runtimes DB does not track the completed process instance (that's
> what
> > > the
> > > > JDBCProcessInstances warn is telling us), but DataIndex, as expected,
> > is
> > > > tracking it in processes and nodes table. And yes it will grow over
> > time.
> > > > What we need is some configurable purge mechanism for DataIndex, so
> it
> > > > eventually removes older completed process instances.
> > > >
> > > > On Tue, Feb 13, 2024 at 12:59 PM Francisco Javier Tirado Sarti <
> > > > ftirados@redhat.com> wrote:
> > > >
> > > > > Hi Martin,
> > > > > Good catch!. Looks like the skipping performed for process
> instances
> > is
> > > > > not applied to node instances. Something we definitely need to
> review
> > > on
> > > > > the runtimes side.
> > > > >
> > > > > On Mon, Feb 12, 2024 at 11:59 PM Martin Weiler
> > <mweiler@ibm.com.invalid
> > > >
> > > > > wrote:
> > > > >
> > > > >> On a somewhat related note, testing a simple workflow (start ->
> > script
> > > > >> node -> end), I see the following messages in the logs:
> > > > >> 2024-02-12 22:49:50,493 28758dde544c WARN
> > > > >> [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> > > > >> (executor-thread-3) Skipping create of process instance id:
> > > > >> 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2
> > > > >>
> > > > >> So far, so good. And I'd expect to see no trace of this process in
> > the
> > > > >> database if I don't have data audit enabled.
> > > > >>
> > > > >> However, the 'processes' table contains a row with state=2, with
> > > related
> > > > >> entries in the 'nodes' table. In a load test, I see these tables
> > grow
> > > > >> significantly over time. Am I missing something to have these
> > entries
> > > > >> cleaned up automatically?
> > > > >>
> > > > >> ________________________________________
> > > > >> From: Martin Weiler <mw...@ibm.com.INVALID>
> > > > >> Sent: Monday, February 12, 2024 3:40 PM
> > > > >> To: dev@kie.apache.org
> > > > >> Subject: [EXTERNAL] RE: [DISCUSSION] Performance issues with
> > > data-index
> > > > >> persistence addon
> > > > >>
> > > > >> Thanks everyone for your input. Based on this discussion, I opened
> > the
> > > > >> following PR:
> > > > >> https://github.com/apache/incubator-kie-kogito-apps/pull/1985
> > > > >>
> > > > >> With this change, the performance seems to be stable over time:
> > > > >>
> > > > >>
> > > >
> > >
> >
> https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
> > > > >>
> > > > >> Martin
> > > > >>
> > > > >> ________________________________________
> > > > >> From: Gonzalo Muñoz <gm...@apache.org>
> > > > >> Sent: Friday, February 9, 2024 9:42 AM
> > > > >> To: dev@kie.apache.org
> > > > >> Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > > data-index
> > > > >> persistence addon
> > > > >>
> > > > >> Great work Francisco,
> > > > >> Martin, take a look at this link with some related tips (in case
> you
> > > > find
> > > > >> it useful):
> > > > >> https://www.cybertec-postgresql.com/en/index-your-foreign-key/
> > > > >>
> > > > >> El vie, 9 feb 2024 a las 17:20, Francisco Javier Tirado Sarti (<
> > > > >> ftirados@redhat.com>) escribió:
> > > > >>
> > > > >> > For the moment being, we will keep JPA till we exhaust all
> > > > >> possibilities,
> > > > >> > let's call switching from jpa to jdbc our hidden plan B ;)
> > > > >> > I already told Martin, but in order everyone to know, just after
> > > > writing
> > > > >> > the previous email, I thought "what if Postgres is not
> > automatically
> > > > >> > indexing foreign keys like mysql?" and, eureka
> > > > >> > Postgres doc
> > > > >> > https://www.postgresql.org/docs/current/ddl-constraints.html
> > > > >> > Mysql doc
> > > > >> >
> > https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> > > > >> > These are the relevant excerpt
> > > > >> >
> > > > >> > *Postgresql*
> > > > >> > *A foreign key must reference columns that either are a primary
> > key
> > > or
> > > > >> form
> > > > >> > a unique constraint, or are columns from a non-partial unique
> > index.
> > > > >> This
> > > > >> > means that the referenced columns always have an index to allow
> > > > >> efficient
> > > > >> > lookups on whether a referencing row has a match. Since a DELETE
> > of
> > > a
> > > > >> row
> > > > >> > from the referenced table or an UPDATE of a referenced column
> will
> > > > >> require
> > > > >> > a scan of the referencing table for rows matching the old value,
> > it
> > > is
> > > > >> > often a good idea to index the referencing columns too. Because
> > this
> > > > is
> > > > >> not
> > > > >> > always needed, and there are many choices available on how to
> > index,
> > > > the
> > > > >> > declaration of a foreign key constraint does not automatically
> > > create
> > > > an
> > > > >> > index on the referencing columns.*
> > > > >> > *Mysql*
> > > > >> > *MySQL requires that foreign key columns be indexed; if you
> > create a
> > > > >> table
> > > > >> > with a foreign key constraint but no index on a given column, an
> > > index
> > > > >> is
> > > > >> > created. *
> > > > >> >
> > > > >> > So I asked Martin to especially create an index for
> > > > process_instance_id
> > > > >> > column on nodes table
> > > > >> > I think that will fix the problem detected on the thread dump.
> > > > >> > The simpler process test to verify queries are fine still
> stands,
> > > > >> though ;)
> > > > >> >
> > > > >> >
> > > > >> > On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <
> tzimanyi@apache.org
> > >
> > > > >> wrote:
> > > > >> >
> > > > >> > > I always preferred pure JDBC over Hibernate myself, just for
> the
> > > > sake
> > > > >> of
> > > > >> > > control of what is happening :) So I would not -1 that myself.
> > > > >> > >
> > > > >> > > Tibor
> > > > >> > >
> > > > >> > > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado Sarti <
> > > > >> > > ftirados@redhat.com>
> > > > >> > > napísal(a):
> > > > >> > >
> > > > >> > > > Hi,
> > > > >> > > > Usually I do not want to talk about work in progress because
> > > > >> > preliminary
> > > > >> > > > conclusions are pretty volatile but, well, there are a
> couple
> > of
> > > > >> things
> > > > >> > > > that can be concluded from the really valuable information
> > that
> > > > >> Martin
> > > > >> > > > provided.
> > > > >> > > > 1) In order to be able to determine if the number of
> > statements
> > > is
> > > > >> > larger
> > > > >> > > > than expected, I asked Martin to test with a simpler process
> > > > >> > definition.
> > > > >> > > > One with just three nodes: start, script and end. The script
> > one
> > > > >> should
> > > > >> > > > change just one variable. This way we can analyze if the
> > number
> > > of
> > > > >> > > queries
> > > > >> > > > is the expected one. From the single log (audit was
> activated
> > > > them)
> > > > >> my
> > > > >> > > > conclusion is that the number of insert/updates over
> processes
> > > and
> > > > >> > nodes
> > > > >> > > > (there a lot over task, that I will prefer to skip for now,
> > baby
> > > > >> steps)
> > > > >> > > is
> > > > >> > > > the expected one.
> > > > >> > > > 2) Analysing the thread dump, we see around 15 threads
> > executing
> > > > >> this
> > > > >> > > line
> > > > >> > > > at
> > > > >> > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > > >> > > > so its pretty clear the code to be optimized ;). I'm
> > evaluating
> > > > >> > > > possibilities within JPA/Hibernate, but I'm starting to
> think
> > > that
> > > > >> it
> > > > >> > > might
> > > > >> > > > be better to switch to JDBC and skip hibernate. Our lives
> will
> > > be
> > > > >> > > simpler,
> > > > >> > > > especially with a schema relatively simple like ours (that
> > will
> > > be
> > > > >> my
> > > > >> > > > recommendation if I was an external consultant)
> > > > >> > > >
> > > > >> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <
> > > tzimanyi@apache.org
> > > > >
> > > > >> > > wrote:
> > > > >> > > >
> > > > >> > > > > Hi,
> > > > >> > > > >
> > > > >> > > > > this will be a bit off-topic. However as far as
> > performance, I
> > > > >> think
> > > > >> > we
> > > > >> > > > > should think about that we have string primary keys
> (IDs). I
> > > > would
> > > > >> > > expect
> > > > >> > > > > the database systems are much better with indexing numeric
> > > keys
> > > > >> than
> > > > >> > > > > strings. I remember from the past, when I was working with
> > > DBs,
> > > > >> that
> > > > >> > > > using
> > > > >> > > > > strings as keys or indexes was a discouraged practice.
> > > > >> > > > >
> > > > >> > > > > Best regards,
> > > > >> > > > > Tibor
> > > > >> > > > >
> > > > >> > > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler
> > > <mweiler@ibm.com.invalid
> > > > >
> > > > >> > > > > napísal(a):
> > > > >> > > > >
> > > > >> > > > > > I changed the test to use MongoDB [1] and I don't see a
> > > > >> performance
> > > > >> > > > > > degradation with this setup [2].
> > > > >> > > > > >
> > > > >> > > > > > Please keep us posted of your findings. Thanks!
> > > > >> > > > > >
> > > > >> > > > > > Martin
> > > > >> > > > > >
> > > > >> > > > > > [1]
> > > > >> > > > >
> > > > >> >
> > > >
> https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > >> > > > > > [2]
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > >> > > > > >
> > > > >> > > > > > ________________________________________
> > > > >> > > > > > From: Francisco Javier Tirado Sarti <
> ftirados@redhat.com>
> > > > >> > > > > > Sent: Wednesday, February 7, 2024 11:40 AM
> > > > >> > > > > > To: dev@kie.apache.org
> > > > >> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues
> > with
> > > > >> > > data-index
> > > > >> > > > > > persistence addon
> > > > >> > > > > >
> > > > >> > > > > > yes, it can be index degradation because of size, but I
> > > > believe
> > > > >> (I
> > > > >> > > > might
> > > > >> > > > > be
> > > > >> > > > > > wrong) the db is too small (yet) for that.
> > > > >> > > > > > But, eventually, Postgres, when the DB is huge enough,
> > > > >> unavoidably
> > > > >> > > will
> > > > >> > > > > > behave like the graphic that Martin sent.
> > > > >> > > > > > Since I believe we are not huge enough (yet), lets rule
> > out
> > > > >> another
> > > > >> > > > issue
> > > > >> > > > > > by analysing the sql logs (I requested those to Martin
> > > offline
> > > > >> and
> > > > >> > he
> > > > >> > > > is
> > > > >> > > > > > going to kindly collect them).
> > > > >> > > > > > Also Im curious to know if Mongo behave in the same way.
> > > > >> > > > > >
> > > > >> > > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez
> Martinez <
> > > > >> > > > > > egonzalez@apache.org> wrote:
> > > > >> > > > > >
> > > > >> > > > > > > Hi Francisco,
> > > > >> > > > > > > I would highly recommend to check indexes and how the
> > > > updates
> > > > >> > work
> > > > >> > > in
> > > > >> > > > > > data
> > > > >> > > > > > > index to avoid full scan table and lock the full
> table.
> > > Some
> > > > >> db
> > > > >> > are
> > > > >> > > > > very
> > > > >> > > > > > > sensitive to that.
> > > > >> > > > > > >
> > > > >> > > > > > > El mié, 7 feb 2024, 18:41, Francisco Javier Tirado
> > Sarti <
> > > > >> > > > > > > ftirados@redhat.com> escribió:
> > > > >> > > > > > >
> > > > >> > > > > > > > Hi Martin,
> > > > >> > > > > > > > While I analyze the data, let me ask you if it is
> > > possible
> > > > >> to
> > > > >> > > > perform
> > > > >> > > > > > > > another check (similar in a way to disabling
> > data-index
> > > > like
> > > > >> > you
> > > > >> > > > do)
> > > > >> > > > > > Can
> > > > >> > > > > > > > you switch to MongoDB persistence and check if the
> > same
> > > > >> > > degradation
> > > > >> > > > > > that
> > > > >> > > > > > > is
> > > > >> > > > > > > > there for postgres remains?
> > > > >> > > > > > > > I do not know if this is feasible but will certainly
> > > > >> indicate
> > > > >> > the
> > > > >> > > > > > problem
> > > > >> > > > > > > > is on the postgres storage layer and I do not have a
> > > clear
> > > > >> > > > prediction
> > > > >> > > > > > of
> > > > >> > > > > > > > what we will see when doing this switch.
> > > > >> > > > > > > >
> > > > >> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler
> > > > >> > > > <mweiler@ibm.com.invalid
> > > > >> > > > > >
> > > > >> > > > > > > > wrote:
> > > > >> > > > > > > >
> > > > >> > > > > > > > > Hi Francisco,
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > thanks for your work on this important topic!
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > I would like to share some test results here,
> which
> > > > might
> > > > >> > help
> > > > >> > > to
> > > > >> > > > > > > improve
> > > > >> > > > > > > > > the codebase even further. I am using the jmeter
> > based
> > > > >> test
> > > > >> > > case
> > > > >> > > > > from
> > > > >> > > > > > > > Pere
> > > > >> > > > > > > > > and Enrique (thanks guys!) [1] which uses a load
> of
> > 30
> > > > >> > threads
> > > > >> > > to
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > 1) start a new process instance (POST)
> > > > >> > > > > > > > > 2) retrieve tasks for a user (GET)
> > > > >> > > > > > > > > 3) fetches task details (GET)
> > > > >> > > > > > > > > 4) complete a task (POST)
> > > > >> > > > > > > > > 5) execute a query on data-audit
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > With this test setup, I noticed that the
> performance
> > > for
> > > > >> the
> > > > >> > > POST
> > > > >> > > > > > > > > requests, in particular the one to start a new
> > process
> > > > >> > > instance,
> > > > >> > > > > > > degrades
> > > > >> > > > > > > > > over time - see graph [2]. If I run the same test
> > > > without
> > > > >> > > > > data-index,
> > > > >> > > > > > > > then
> > > > >> > > > > > > > > there is no such performance degradation [3]. You
> > can
> > > > >> find a
> > > > >> > > > thread
> > > > >> > > > > > > dump
> > > > >> > > > > > > > > captured a few minutes into the first test here
> [4]
> > > that
> > > > >> > might
> > > > >> > > > help
> > > > >> > > > > > to
> > > > >> > > > > > > > see
> > > > >> > > > > > > > > some of the contention points.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > I'd appreciate if you could take a look and see if
> > > there
> > > > >> is
> > > > >> > > > > something
> > > > >> > > > > > > > that
> > > > >> > > > > > > > > can be further improved based on your previous
> work.
> > > If
> > > > >> you
> > > > >> > > need
> > > > >> > > > > any
> > > > >> > > > > > > > > additional data, let me know, but otherwise it is
> > > > >> > > straightforward
> > > > >> > > > > to
> > > > >> > > > > > > run
> > > > >> > > > > > > > > the jmeter test as well.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Thanks,
> > > > >> > > > > > > > > Martin
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > [1]
> > > > >> https://github.com/pefernan/job-service-refactor-test/
> > > > >> > > > > > > > > [2]
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > >> > > > > > > > > [3]
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > >> > > > > > > > > [4]
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > ________________________________________
> > > > >> > > > > > > > > From: Francisco Javier Tirado Sarti <
> > > > ftirados@redhat.com>
> > > > >> > > > > > > > > Sent: Wednesday, January 17, 2024 9:13 AM
> > > > >> > > > > > > > > To: dev@kie.apache.org
> > > > >> > > > > > > > > Cc: Pere Fernandez Perez
> > > > >> > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance
> > > issues
> > > > >> with
> > > > >> > > > > > data-index
> > > > >> > > > > > > > > persistence addon
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Hi Alex,
> > > > >> > > > > > > > > I did not take times (which depends on a number of
> > > > >> variables
> > > > >> > > that
> > > > >> > > > > > > > > drastically change between environments), but
> verify
> > > > that
> > > > >> the
> > > > >> > > > > number
> > > > >> > > > > > of
> > > > >> > > > > > > > > updates has been reduced drastically without
> losing
> > > > >> > > > functionality,
> > > > >> > > > > > > which
> > > > >> > > > > > > > is
> > > > >> > > > > > > > > objectively a good thing. If before the change,
> for
> > > > every
> > > > >> > node
> > > > >> > > > > > > executed,
> > > > >> > > > > > > > we
> > > > >> > > > > > > > > have an update for every node previously executed,
> > so
> > > > if a
> > > > >> > > > process
> > > > >> > > > > > have
> > > > >> > > > > > > > 50
> > > > >> > > > > > > > > nodes to execute, we were performing nearly
> 50*51/2
> > > > >> updates,
> > > > >> > > > which
> > > > >> > > > > > > gives
> > > > >> > > > > > > > us
> > > > >> > > > > > > > > a total of  1275 updates, now we have just one for
> > > every
> > > > >> node
> > > > >> > > > being
> > > > >> > > > > > > > > executed, implying a total of 50 updates.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli <
> > > > >> > > alex@porcelli.me>
> > > > >> > > > > > > wrote:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > Francisco,
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > I noticed that your PR has been merged, but I
> was
> > > > >> expecting
> > > > >> > > (at
> > > > >> > > > > > least
> > > > >> > > > > > > > > > was my understanding from this thread) that
> before
> > > > >> merging
> > > > >> > > some
> > > > >> > > > > > > > > > benchmark data would be shared in advance - to
> > > assess
> > > > >> the
> > > > >> > > > > > > cost/benefit
> > > > >> > > > > > > > > > of such a decent size change.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Do you have any information to share?
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM Francisco Javier
> > > > Tirado
> > > > >> > Sarti
> > > > >> > > > > > > > > > <ft...@redhat.com> wrote:
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Yes, as intended, now we have one select and
> one
> > > > >> > > > insert/update
> > > > >> > > > > > per
> > > > >> > > > > > > > node
> > > > >> > > > > > > > > > > event.
> > > > >> > > > > > > > > > > I moved the PR as ready for review and give
> > @Pere
> > > > >> > Fernandez
> > > > >> > > > > Perez
> > > > >> > > > > > > > > > > <pe...@redhat.com> permission to the
> branch
> > so
> > > > he
> > > > >> can
> > > > >> > > > edit
> > > > >> > > > > it
> > > > >> > > > > > > in
> > > > >> > > > > > > > > the
> > > > >> > > > > > > > > > > next two weeks (Ill be on PTO)  if desired,
> > before
> > > > >> > merging.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex Porcelli
> <
> > > > >> > > > > alex@porcelli.me>
> > > > >> > > > > > > > > wrote:
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > > Cool, thank you Francisco!
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > Did you manage to get some preliminary data
> > > about
> > > > >> > > > > improvements?
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM Francisco
> > > Javier
> > > > >> > Tirado
> > > > >> > > > > Sarti
> > > > >> > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Yes, after some delay because of quarkus 3
> > > > >> migration.
> > > > >> > > Im
> > > > >> > > > > > > refining
> > > > >> > > > > > > > > > this
> > > > >> > > > > > > > > > > > > draft PR
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > >
> > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex
> > Porcelli
> > > <
> > > > >> > > > > > > alex@porcelli.me>
> > > > >> > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Any update or new findings on this
> topic?
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM
> Francisco
> > > > Javier
> > > > >> > > Tirado
> > > > >> > > > > > Sarti
> > > > >> > > > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > Hi Alex,
> > > > >> > > > > > > > > > > > > > > After considering different options to
> > > > improve
> > > > >> > > > > > performance,
> > > > >> > > > > > > > we
> > > > >> > > > > > > > > > feel
> > > > >> > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > > > it
> > > > >> > > > > > > > > > > > > > > is time to "partially" move away from
> > the
> > > > >> current
> > > > >> > > Map
> > > > >> > > > > > style
> > > > >> > > > > > > > > > > > interface (
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > >> > > > > > > > > > > > > > )
> > > > >> > > > > > > > > > > > > > > which was shared with Trusty, to one
> > more
> > > > >> > suitable
> > > > >> > > > for
> > > > >> > > > > > > usage
> > > > >> > > > > > > > > > with a
> > > > >> > > > > > > > > > > > > > > relational DB like postgresql (but
> still
> > > > >> > compatible
> > > > >> > > > > with
> > > > >> > > > > > > big
> > > > >> > > > > > > > > > table
> > > > >> > > > > > > > > > > > dbs).
> > > > >> > > > > > > > > > > > > > > The idea will be to replace generic
> > > Storage
> > > > >> > > interface
> > > > >> > > > > by
> > > > >> > > > > > > four
> > > > >> > > > > > > > > > > > specific
> > > > >> > > > > > > > > > > > > > > interfaces (which will inherit from a
> > > common
> > > > >> one
> > > > >> > > that
> > > > >> > > > > > keeps
> > > > >> > > > > > > > the
> > > > >> > > > > > > > > > query
> > > > >> > > > > > > > > > > > > > part
> > > > >> > > > > > > > > > > > > > > at is it. with get and query methods),
> > > that
> > > > >> will
> > > > >> > > > > include
> > > > >> > > > > > > the
> > > > >> > > > > > > > > > required
> > > > >> > > > > > > > > > > > > > > modification operations for the four
> > > > DataIndex
> > > > >> > > > > "domains":
> > > > >> > > > > > > > > > > > > > processinstance,
> > > > >> > > > > > > > > > > > > > > usertask, processdefinitions and jobs.
> > > Those
> > > > >> > > > interfaces
> > > > >> > > > > > > will
> > > > >> > > > > > > > > > define
> > > > >> > > > > > > > > > > > > > methods
> > > > >> > > > > > > > > > > > > > > like addNode, addVariable, updateTask,
> > > > >> > > > > addAttachment.....
> > > > >> > > > > > > > that
> > > > >> > > > > > > > > > will
> > > > >> > > > > > > > > > > > allow
> > > > >> > > > > > > > > > > > > > > the persistent layer implementation
> to
> > > just
> > > > >> > update
> > > > >> > > > the
> > > > >> > > > > > > > needed
> > > > >> > > > > > > > > > info
> > > > >> > > > > > > > > > > > in
> > > > >> > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > DB  (for example, for addNode in
> > Postgres,
> > > > >> just
> > > > >> > > > insert
> > > > >> > > > > a
> > > > >> > > > > > > row
> > > > >> > > > > > > > > into
> > > > >> > > > > > > > > > > > nodes
> > > > >> > > > > > > > > > > > > > > table, for addNode in Mongo, basically
> > the
> > > > >> same
> > > > >> > > > atomic
> > > > >> > > > > > > upsert
> > > > >> > > > > > > > > > > > operation
> > > > >> > > > > > > > > > > > > > > that is currently done). Therefore, we
> > > > >> increase
> > > > >> > > > > > performance
> > > > >> > > > > > > > for
> > > > >> > > > > > > > > > > > Postgres
> > > > >> > > > > > > > > > > > > > > and keep the current one for Mongo.
> The
> > > > >> current
> > > > >> > DB
> > > > >> > > > > > schemas
> > > > >> > > > > > > > > won't
> > > > >> > > > > > > > > > be
> > > > >> > > > > > > > > > > > > > > touched.
> > > > >> > > > > > > > > > > > > > > Since the code change is large, I do
> not
> > > > think
> > > > >> > I'll
> > > > >> > > > be
> > > > >> > > > > > able
> > > > >> > > > > > > > to
> > > > >> > > > > > > > > > have
> > > > >> > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > PR
> > > > >> > > > > > > > > > > > > > > ready till next week.
> > > > >> > > > > > > > > > > > > > > But before starting, please let me
> know
> > if
> > > > >> that
> > > > >> > > > > approach
> > > > >> > > > > > is
> > > > >> > > > > > > > > fine
> > > > >> > > > > > > > > > for
> > > > >> > > > > > > > > > > > you.
> > > > >> > > > > > > > > > > > > > > Best regards.
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 6:55 PM Alex
> > > > Porcelli
> > > > >> <
> > > > >> > > > > > > > > alex@porcelli.me>
> > > > >> > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > Thank you Francisco to getting
> deeper
> > on
> > > > >> this…
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > Looking forward to see the results
> of
> > > your
> > > > >> > > > suggested
> > > > >> > > > > > > > > > improvements.
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 9:40 AM
> > > Francisco
> > > > >> > Javier
> > > > >> > > > > Tirado
> > > > >> > > > > > > > > Sarti <
> > > > >> > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > I forgot to attach the queries
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 3:04 PM
> > > > Francisco
> > > > >> > > Javier
> > > > >> > > > > > Tirado
> > > > >> > > > > > > > > > Sarti <
> > > > >> > > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > > >> Hi,
> > > > >> > > > > > > > > > > > > > > > >> A brief update on this topic.
> > > > >> > > > > > > > > > > > > > > > >> After doing a simple test with
> > > example
> > > > >> > > > > > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > >> > > > > > > > > > > > > > > > ,
> > > > >> > > > > > > > > > > > > > > > >> the number of updates over Nodes
> > > table
> > > > is
> > > > >> > n*n,
> > > > >> > > > so
> > > > >> > > > > we
> > > > >> > > > > > > > > manage
> > > > >> > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > obtain a
> > > > >> > > > > > > > > > > > > > > > >> perfect quadratic performance
> > > > >> degradation.
> > > > >> > The
> > > > >> > > > > > problem
> > > > >> > > > > > > > is
> > > > >> > > > > > > > > > worse
> > > > >> > > > > > > > > > > > in
> > > > >> > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > case
> > > > >> > > > > > > > > > > > > > > > >> of Serverless Workflow than in
> BPMN
> > > > >> because
> > > > >> > we
> > > > >> > > > the
> > > > >> > > > > > > > number
> > > > >> > > > > > > > > of
> > > > >> > > > > > > > > > > > nodes
> > > > >> > > > > > > > > > > > > > is
> > > > >> > > > > > > > > > > > > > > > >> greater than the number of
> states.
> > In
> > > > >> that
> > > > >> > > > > example N
> > > > >> > > > > > > is
> > > > >> > > > > > > > > 16,
> > > > >> > > > > > > > > > but
> > > > >> > > > > > > > > > > > for
> > > > >> > > > > > > > > > > > > > a
> > > > >> > > > > > > > > > > > > > > > more
> > > > >> > > > > > > > > > > > > > > > >> complex workflow it would be
> > > certainly
> > > > >> > large.
> > > > >> > > > > > > > > > > > > > > > >> I think that this is more related
> > to
> > > > how
> > > > >> we
> > > > >> > > are
> > > > >> > > > > > > handling
> > > > >> > > > > > > > > > JPA in
> > > > >> > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > > code,
> > > > >> > > > > > > > > > > > > > > > >> in particular the mapping from
> > model
> > > to
> > > > >> > entity
> > > > >> > > > > > > > (basically
> > > > >> > > > > > > > > > JPA is
> > > > >> > > > > > > > > > > > > > blind
> > > > >> > > > > > > > > > > > > > > > and
> > > > >> > > > > > > > > > > > > > > > >> has to update all nodes for every
> > > write
> > > > >> > > because
> > > > >> > > > it
> > > > >> > > > > > > > > believes
> > > > >> > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > node has
> > > > >> > > > > > > > > > > > > > > > >> been updated, although it is not)
> > > than
> > > > an
> > > > >> > > issue
> > > > >> > > > in
> > > > >> > > > > > the
> > > > >> > > > > > > > > table
> > > > >> > > > > > > > > > > > > > definition.
> > > > >> > > > > > > > > > > > > > > > >> In fact, when using JPA,
> separating
> > > the
> > > > >> > server
> > > > >> > > > > model
> > > > >> > > > > > > > from
> > > > >> > > > > > > > > > the
> > > > >> > > > > > > > > > > > JPA
> > > > >> > > > > > > > > > > > > > > > entity is
> > > > >> > > > > > > > > > > > > > > > >> not a good idea, especially if
> the
> > > > entity
> > > > >> > > > contains
> > > > >> > > > > > > > > > collections.
> > > > >> > > > > > > > > > > > I
> > > > >> > > > > > > > > > > > > > will
> > > > >> > > > > > > > > > > > > > > > try
> > > > >> > > > > > > > > > > > > > > > >> to change that without breaking
> > > > anything.
> > > > >> > > > > > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > > > > > >> On Wed, Nov 22, 2023 at 12:10 PM
> > > > Enrique
> > > > >> > > > Gonzalez
> > > > >> > > > > > > > > Martinez <
> > > > >> > > > > > > > > > > > > > > > >> egonzalez@apache.org> wrote:
> > > > >> > > > > > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > > > > > >>> After the events split you now
> > will
> > > > >> need to
> > > > >> > > > > create
> > > > >> > > > > > a
> > > > >> > > > > > > > node
> > > > >> > > > > > > > > > > > instance
> > > > >> > > > > > > > > > > > > > > > >>> model instance of making
> > independent
> > > > >> from
> > > > >> > the
> > > > >> > > > > > process
> > > > >> > > > > > > > > > instance.
> > > > >> > > > > > > > > > > > > > > > >>> That should do the trick.
> > > > >> > > > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > > > >>> Regarding deleting/inserting it
> > was
> > > > >> fixed
> > > > >> > at
> > > > >> > > > some
> > > > >> > > > > > > > point.
> > > > >> > > > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > > > >>> El mar, 21 nov 2023 a las 20:22,
> > > > >> Francisco
> > > > >> > > > Javier
> > > > >> > > > > > > > Tirado
> > > > >> > > > > > > > > > Sarti
> > > > >> > > > > > > > > > > > > > > > >>> (<ft...@redhat.com>)
> escribió:
> > > > >> > > > > > > > > > > > > > > > >>> >
> > > > >> > > > > > > > > > > > > > > > >>> > Hi Martin,
> > > > >> > > > > > > > > > > > > > > > >>> > I have a task to review
> > > performance
> > > > of
> > > > >> > > > > > > > > > > > > > > > >>> >
> > > > >> > > > > > > > > > > > > > > > >>> >
> > ProcessInstanceNodeDataEventMerger
> > > > >> > > > > > > > > > > > > > > > >>> > My idea is to reduce the
> number
> > of
> > > > >> delete
> > > > >> > > > > inserts
> > > > >> > > > > > > > when
> > > > >> > > > > > > > > > > > processing
> > > > >> > > > > > > > > > > > > > > > >>> events
> > > > >> > > > > > > > > > > > > > > > >>> > and try to do it incremental.
> > > > >> > > > > > > > > > > > > > > > >>> > That should improve
> performance.
> > > > >> > > > > > > > > > > > > > > > >>> > PS:
> > > > >> > > > > > > > > > > > > > > > >>> > I was planning to send an
> e-mail
> > > > >> tomorrow
> > > > >> > > > > > > announcing
> > > > >> > > > > > > > > > that in
> > > > >> > > > > > > > > > > > > > case you
> > > > >> > > > > > > > > > > > > > > > >>> were
> > > > >> > > > > > > > > > > > > > > > >>> > already working on a fix for
> > > that. I
> > > > >> > assume
> > > > >> > > > you
> > > > >> > > > > > are
> > > > >> > > > > > > > not
> > > > >> > > > > > > > > > and I
> > > > >> > > > > > > > > > > > > > would
> > > > >> > > > > > > > > > > > > > > > be
> > > > >> > > > > > > > > > > > > > > > >>> > sending a PR soon.
> > > > >> > > > > > > > > > > > > > > > >>> >
> > > > >> > > > > > > > > > > > > > > > >>> > On Tue, Nov 21, 2023 at
> 6:09 PM
> > > > Martin
> > > > >> > > Weiler
> > > > >> > > > > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > >> > > > > > > > > > > > > > > > >>> >
> > > > >> > > > > > > > > > > > > > > > >>> > wrote:
> > > > >> > > > > > > > > > > > > > > > >>> >
> > > > >> > > > > > > > > > > > > > > > >>> > > I looked into the new
> examples
> > > > using
> > > > >> > > > > data-index
> > > > >> > > > > > > > > > persistence
> > > > >> > > > > > > > > > > > > > addon -
> > > > >> > > > > > > > > > > > > > > > >>> Neus'
> > > > >> > > > > > > > > > > > > > > > >>> > > PR#1813 [1] for serverless
> and
> > > > >> Pere's
> > > > >> > > > branch
> > > > >> > > > > > [2]
> > > > >> > > > > > > > for
> > > > >> > > > > > > > > > > > workflow
> > > > >> > > > > > > > > > > > > > > > (great
> > > > >> > > > > > > > > > > > > > > > >>> job
> > > > >> > > > > > > > > > > > > > > > >>> > > both!) - and they work
> without
> > > > >> issues
> > > > >> > > using
> > > > >> > > > > > > single
> > > > >> > > > > > > > > > > > requests.
> > > > >> > > > > > > > > > > > > > > > >>> However, under
> > > > >> > > > > > > > > > > > > > > > >>> > > some load (I used 'ab' for
> > > testing
> > > > >> > with a
> > > > >> > > > > light
> > > > >> > > > > > > > > > > > concurrency of
> > > > >> > > > > > > > > > > > > > 10
> > > > >> > > > > > > > > > > > > > > > >>> parallel
> > > > >> > > > > > > > > > > > > > > > >>> > > requests) I ran into the
> > > following
> > > > >> > > > problems:
> > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > >> > > > > > > > > > > > > > > > >>> > > (1) Large number of
> > > insert/delete
> > > > >> calls
> > > > >> > > > (eg.
> > > > >> > > > > > for
> > > > >> > > > > > > > > tables
> > > > >> > > > > > > > > > > > such as
> > > > >> > > > > > > > > > > > > > > > >>> nodes,
> > > > >> > > > > > > > > > > > > > > > >>> > > definitions, etc)
> > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > >> > > > > > > > > > > > > > > > >>> > > (2) Hibernate
> > > > >> OptimisticLockExceptions
> > > > >> > /
> > > > >> > > > > > > > > > > > StaleStateExceptions
> > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > >> > > > > > > > > > > > > > > > >>> > > (3) DB deadlocks
> > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > >> > > > > > > > > > > > > > > > >>> > > (4) Error responses, slow
> > > response
> > > > >> > times
> > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > >> > > > > > > > > > > > > > > > >>> > > The reason I am reaching out
> > > with
> > > > >> this
> > > > >> > > > topic
> > > > >> > > > > > here
> > > > >> > > > > > > > is
> > > > >> > > > > > > > > to
> > > > >> > > > > > > > > > > > find
> > > > >> > > > > > > > > > > > > > out if
> > > > >> > > > > > > > > > > > > > > > >>> we are
> > > > >> > > > > > > > > > > > > > > > >>> > > aware of this issue, and if
> > > > someone
> > > > >> is
> > > > >> > > > > already
> > > > >> > > > > > > > > looking
> > > > >> > > > > > > > > > > > into or
> > > > >> > > > > > > > > > > > > > > > being
> > > > >> > > > > > > > > > > > > > > > >>> > > assigned to it?
> > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > >> > > > > > > > > > > > > > > > >>> > > Thanks,
> > > > >> > > > > > > > > > > > > > > > >>> > > Martin
> > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > >> > > > > > > > > > > > > > > > >>> > > [1]
> > > > >> > > > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > >
> > > > >> >
> https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > >> > > > > > > > > > > > > > > > >>> > > [2]
> > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > >> > > > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > >
> > > > >>
> > ---------------------------------------------------------------------
> > > > >> > > > > > > > > > > > > > > > >>> > > To unsubscribe, e-mail:
> > > > >> > > > > > > > > dev-unsubscribe@kie.apache.org
> > > > >> > > > > > > > > > > > > > > > >>> > > For additional commands,
> > e-mail:
> > > > >> > > > > > > > > > dev-help@kie.apache.org
> > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > >> > > > > > > > > > > > > > > > >>> > >
> > > > >> > > > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > >
> > > > >> >
> > > ---------------------------------------------------------------------
> > > > >> > > > > > > > > > > > > > > > >>> To unsubscribe, e-mail:
> > > > >> > > > > > > dev-unsubscribe@kie.apache.org
> > > > >> > > > > > > > > > > > > > > > >>> For additional commands, e-mail:
> > > > >> > > > > > > > dev-help@kie.apache.org
> > > > >> > > > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > >
> > > > >>
> > ---------------------------------------------------------------------
> > > > >> > > > > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > > >> > > > > > dev-unsubscribe@kie.apache.org
> > > > >> > > > > > > > > > > > > > > > > For additional commands, e-mail:
> > > > >> > > > > > > dev-help@kie.apache.org
> > > > >> > > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > >
> > > > >> >
> > > ---------------------------------------------------------------------
> > > > >> > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > > >> > > dev-unsubscribe@kie.apache.org
> > > > >> > > > > > > > > > > > > > For additional commands, e-mail:
> > > > >> > > > dev-help@kie.apache.org
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > >
> > > > >>
> > ---------------------------------------------------------------------
> > > > >> > > > > > > > > > > > To unsubscribe, e-mail:
> > > > >> dev-unsubscribe@kie.apache.org
> > > > >> > > > > > > > > > > > For additional commands, e-mail:
> > > > >> > dev-help@kie.apache.org
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > >
> > > > >> >
> > > ---------------------------------------------------------------------
> > > > >> > > > > > > > > > To unsubscribe, e-mail:
> > > > dev-unsubscribe@kie.apache.org
> > > > >> > > > > > > > > > For additional commands, e-mail:
> > > > >> dev-help@kie.apache.org
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > >
> > > > >>
> > ---------------------------------------------------------------------
> > > > >> > > > > > > > > To unsubscribe, e-mail:
> > > dev-unsubscribe@kie.apache.org
> > > > >> > > > > > > > > For additional commands, e-mail:
> > > > dev-help@kie.apache.org
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >>
> > ---------------------------------------------------------------------
> > > > >> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > >> For additional commands, e-mail: dev-help@kie.apache.org
> > > > >>
> > > > >>
> > > >
> > >
> >
>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Enrique Gonzalez Martinez <eg...@apache.org>]

Alex:
Right now the data index is working in the same way as it did in v7 with
the emitters. The only difference between two impl is that in here the
storage is pgsql instead elastic search.  You are right regarding is a
snapshot of the last state of the process but we did never define how long
would be alive that dats Honestly i am happy right now with the way it
works. The clean up mechanism is still tbd because we still need to discuss
other stuff first.


Regarding stp is to leave no trail because u can get the outcome directly
from the call. It was defined like that in v7. So there is no use for the
index or the audit.

El lun, 19 feb 2024, 14:13, Francisco Javier Tirado Sarti <
ftirados@redhat.com> escribió:

> Hi Alex,
> There has been some confusion about the purpose of DataIndex. To be honest
> I believe they were already sorted out, but your e-mail makes me think that
> is not the case ;). I let Kris to clarify that with you. My view is that
> data-index is a way to query recently closed and active processes (the key
> here is the definition of recently, which in my opinion should be
> configurable)
> But, besides that discussion and being pragmatic, keeping finishing process
> instances "for a while" in DataIndex was the only way for users to query
> the result of straight through processes. That's a function that cannot be
> removed right now
>
> On Mon, Feb 19, 2024 at 1:33 PM Alex Porcelli <po...@apache.org> wrote:
>
> > if data index was supposed to provide snapshot view of the process
> > instance… why do we keep it after the process instance is finished?
> >
> >
> > On Mon, Feb 19, 2024 at 7:12 AM Francisco Javier Tirado Sarti <
> > ftirados@redhat.com> wrote:
> >
> > > Hi Martin.
> > > After taking a deeper look at this, I realize that the behaviour is the
> > > expected one.
> > > Runtimes DB does not track the completed process instance (that's what
> > the
> > > JDBCProcessInstances warn is telling us), but DataIndex, as expected,
> is
> > > tracking it in processes and nodes table. And yes it will grow over
> time.
> > > What we need is some configurable purge mechanism for DataIndex, so it
> > > eventually removes older completed process instances.
> > >
> > > On Tue, Feb 13, 2024 at 12:59 PM Francisco Javier Tirado Sarti <
> > > ftirados@redhat.com> wrote:
> > >
> > > > Hi Martin,
> > > > Good catch!. Looks like the skipping performed for process instances
> is
> > > > not applied to node instances. Something we definitely need to review
> > on
> > > > the runtimes side.
> > > >
> > > > On Mon, Feb 12, 2024 at 11:59 PM Martin Weiler
> <mweiler@ibm.com.invalid
> > >
> > > > wrote:
> > > >
> > > >> On a somewhat related note, testing a simple workflow (start ->
> script
> > > >> node -> end), I see the following messages in the logs:
> > > >> 2024-02-12 22:49:50,493 28758dde544c WARN
> > > >> [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> > > >> (executor-thread-3) Skipping create of process instance id:
> > > >> 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2
> > > >>
> > > >> So far, so good. And I'd expect to see no trace of this process in
> the
> > > >> database if I don't have data audit enabled.
> > > >>
> > > >> However, the 'processes' table contains a row with state=2, with
> > related
> > > >> entries in the 'nodes' table. In a load test, I see these tables
> grow
> > > >> significantly over time. Am I missing something to have these
> entries
> > > >> cleaned up automatically?
> > > >>
> > > >> ________________________________________
> > > >> From: Martin Weiler <mw...@ibm.com.INVALID>
> > > >> Sent: Monday, February 12, 2024 3:40 PM
> > > >> To: dev@kie.apache.org
> > > >> Subject: [EXTERNAL] RE: [DISCUSSION] Performance issues with
> > data-index
> > > >> persistence addon
> > > >>
> > > >> Thanks everyone for your input. Based on this discussion, I opened
> the
> > > >> following PR:
> > > >> https://github.com/apache/incubator-kie-kogito-apps/pull/1985
> > > >>
> > > >> With this change, the performance seems to be stable over time:
> > > >>
> > > >>
> > >
> >
> https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
> > > >>
> > > >> Martin
> > > >>
> > > >> ________________________________________
> > > >> From: Gonzalo Muñoz <gm...@apache.org>
> > > >> Sent: Friday, February 9, 2024 9:42 AM
> > > >> To: dev@kie.apache.org
> > > >> Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > data-index
> > > >> persistence addon
> > > >>
> > > >> Great work Francisco,
> > > >> Martin, take a look at this link with some related tips (in case you
> > > find
> > > >> it useful):
> > > >> https://www.cybertec-postgresql.com/en/index-your-foreign-key/
> > > >>
> > > >> El vie, 9 feb 2024 a las 17:20, Francisco Javier Tirado Sarti (<
> > > >> ftirados@redhat.com>) escribió:
> > > >>
> > > >> > For the moment being, we will keep JPA till we exhaust all
> > > >> possibilities,
> > > >> > let's call switching from jpa to jdbc our hidden plan B ;)
> > > >> > I already told Martin, but in order everyone to know, just after
> > > writing
> > > >> > the previous email, I thought "what if Postgres is not
> automatically
> > > >> > indexing foreign keys like mysql?" and, eureka
> > > >> > Postgres doc
> > > >> > https://www.postgresql.org/docs/current/ddl-constraints.html
> > > >> > Mysql doc
> > > >> >
> https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> > > >> > These are the relevant excerpt
> > > >> >
> > > >> > *Postgresql*
> > > >> > *A foreign key must reference columns that either are a primary
> key
> > or
> > > >> form
> > > >> > a unique constraint, or are columns from a non-partial unique
> index.
> > > >> This
> > > >> > means that the referenced columns always have an index to allow
> > > >> efficient
> > > >> > lookups on whether a referencing row has a match. Since a DELETE
> of
> > a
> > > >> row
> > > >> > from the referenced table or an UPDATE of a referenced column will
> > > >> require
> > > >> > a scan of the referencing table for rows matching the old value,
> it
> > is
> > > >> > often a good idea to index the referencing columns too. Because
> this
> > > is
> > > >> not
> > > >> > always needed, and there are many choices available on how to
> index,
> > > the
> > > >> > declaration of a foreign key constraint does not automatically
> > create
> > > an
> > > >> > index on the referencing columns.*
> > > >> > *Mysql*
> > > >> > *MySQL requires that foreign key columns be indexed; if you
> create a
> > > >> table
> > > >> > with a foreign key constraint but no index on a given column, an
> > index
> > > >> is
> > > >> > created. *
> > > >> >
> > > >> > So I asked Martin to especially create an index for
> > > process_instance_id
> > > >> > column on nodes table
> > > >> > I think that will fix the problem detected on the thread dump.
> > > >> > The simpler process test to verify queries are fine still stands,
> > > >> though ;)
> > > >> >
> > > >> >
> > > >> > On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <tzimanyi@apache.org
> >
> > > >> wrote:
> > > >> >
> > > >> > > I always preferred pure JDBC over Hibernate myself, just for the
> > > sake
> > > >> of
> > > >> > > control of what is happening :) So I would not -1 that myself.
> > > >> > >
> > > >> > > Tibor
> > > >> > >
> > > >> > > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado Sarti <
> > > >> > > ftirados@redhat.com>
> > > >> > > napísal(a):
> > > >> > >
> > > >> > > > Hi,
> > > >> > > > Usually I do not want to talk about work in progress because
> > > >> > preliminary
> > > >> > > > conclusions are pretty volatile but, well, there are a couple
> of
> > > >> things
> > > >> > > > that can be concluded from the really valuable information
> that
> > > >> Martin
> > > >> > > > provided.
> > > >> > > > 1) In order to be able to determine if the number of
> statements
> > is
> > > >> > larger
> > > >> > > > than expected, I asked Martin to test with a simpler process
> > > >> > definition.
> > > >> > > > One with just three nodes: start, script and end. The script
> one
> > > >> should
> > > >> > > > change just one variable. This way we can analyze if the
> number
> > of
> > > >> > > queries
> > > >> > > > is the expected one. From the single log (audit was activated
> > > them)
> > > >> my
> > > >> > > > conclusion is that the number of insert/updates over processes
> > and
> > > >> > nodes
> > > >> > > > (there a lot over task, that I will prefer to skip for now,
> baby
> > > >> steps)
> > > >> > > is
> > > >> > > > the expected one.
> > > >> > > > 2) Analysing the thread dump, we see around 15 threads
> executing
> > > >> this
> > > >> > > line
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > >> > > > so its pretty clear the code to be optimized ;). I'm
> evaluating
> > > >> > > > possibilities within JPA/Hibernate, but I'm starting to think
> > that
> > > >> it
> > > >> > > might
> > > >> > > > be better to switch to JDBC and skip hibernate. Our lives will
> > be
> > > >> > > simpler,
> > > >> > > > especially with a schema relatively simple like ours (that
> will
> > be
> > > >> my
> > > >> > > > recommendation if I was an external consultant)
> > > >> > > >
> > > >> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <
> > tzimanyi@apache.org
> > > >
> > > >> > > wrote:
> > > >> > > >
> > > >> > > > > Hi,
> > > >> > > > >
> > > >> > > > > this will be a bit off-topic. However as far as
> performance, I
> > > >> think
> > > >> > we
> > > >> > > > > should think about that we have string primary keys (IDs). I
> > > would
> > > >> > > expect
> > > >> > > > > the database systems are much better with indexing numeric
> > keys
> > > >> than
> > > >> > > > > strings. I remember from the past, when I was working with
> > DBs,
> > > >> that
> > > >> > > > using
> > > >> > > > > strings as keys or indexes was a discouraged practice.
> > > >> > > > >
> > > >> > > > > Best regards,
> > > >> > > > > Tibor
> > > >> > > > >
> > > >> > > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler
> > <mweiler@ibm.com.invalid
> > > >
> > > >> > > > > napísal(a):
> > > >> > > > >
> > > >> > > > > > I changed the test to use MongoDB [1] and I don't see a
> > > >> performance
> > > >> > > > > > degradation with this setup [2].
> > > >> > > > > >
> > > >> > > > > > Please keep us posted of your findings. Thanks!
> > > >> > > > > >
> > > >> > > > > > Martin
> > > >> > > > > >
> > > >> > > > > > [1]
> > > >> > > > >
> > > >> >
> > > https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > >> > > > > > [2]
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > >> > > > > >
> > > >> > > > > > ________________________________________
> > > >> > > > > > From: Francisco Javier Tirado Sarti <ft...@redhat.com>
> > > >> > > > > > Sent: Wednesday, February 7, 2024 11:40 AM
> > > >> > > > > > To: dev@kie.apache.org
> > > >> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues
> with
> > > >> > > data-index
> > > >> > > > > > persistence addon
> > > >> > > > > >
> > > >> > > > > > yes, it can be index degradation because of size, but I
> > > believe
> > > >> (I
> > > >> > > > might
> > > >> > > > > be
> > > >> > > > > > wrong) the db is too small (yet) for that.
> > > >> > > > > > But, eventually, Postgres, when the DB is huge enough,
> > > >> unavoidably
> > > >> > > will
> > > >> > > > > > behave like the graphic that Martin sent.
> > > >> > > > > > Since I believe we are not huge enough (yet), lets rule
> out
> > > >> another
> > > >> > > > issue
> > > >> > > > > > by analysing the sql logs (I requested those to Martin
> > offline
> > > >> and
> > > >> > he
> > > >> > > > is
> > > >> > > > > > going to kindly collect them).
> > > >> > > > > > Also Im curious to know if Mongo behave in the same way.
> > > >> > > > > >
> > > >> > > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez Martinez <
> > > >> > > > > > egonzalez@apache.org> wrote:
> > > >> > > > > >
> > > >> > > > > > > Hi Francisco,
> > > >> > > > > > > I would highly recommend to check indexes and how the
> > > updates
> > > >> > work
> > > >> > > in
> > > >> > > > > > data
> > > >> > > > > > > index to avoid full scan table and lock the full table.
> > Some
> > > >> db
> > > >> > are
> > > >> > > > > very
> > > >> > > > > > > sensitive to that.
> > > >> > > > > > >
> > > >> > > > > > > El mié, 7 feb 2024, 18:41, Francisco Javier Tirado
> Sarti <
> > > >> > > > > > > ftirados@redhat.com> escribió:
> > > >> > > > > > >
> > > >> > > > > > > > Hi Martin,
> > > >> > > > > > > > While I analyze the data, let me ask you if it is
> > possible
> > > >> to
> > > >> > > > perform
> > > >> > > > > > > > another check (similar in a way to disabling
> data-index
> > > like
> > > >> > you
> > > >> > > > do)
> > > >> > > > > > Can
> > > >> > > > > > > > you switch to MongoDB persistence and check if the
> same
> > > >> > > degradation
> > > >> > > > > > that
> > > >> > > > > > > is
> > > >> > > > > > > > there for postgres remains?
> > > >> > > > > > > > I do not know if this is feasible but will certainly
> > > >> indicate
> > > >> > the
> > > >> > > > > > problem
> > > >> > > > > > > > is on the postgres storage layer and I do not have a
> > clear
> > > >> > > > prediction
> > > >> > > > > > of
> > > >> > > > > > > > what we will see when doing this switch.
> > > >> > > > > > > >
> > > >> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler
> > > >> > > > <mweiler@ibm.com.invalid
> > > >> > > > > >
> > > >> > > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > Hi Francisco,
> > > >> > > > > > > > >
> > > >> > > > > > > > > thanks for your work on this important topic!
> > > >> > > > > > > > >
> > > >> > > > > > > > > I would like to share some test results here, which
> > > might
> > > >> > help
> > > >> > > to
> > > >> > > > > > > improve
> > > >> > > > > > > > > the codebase even further. I am using the jmeter
> based
> > > >> test
> > > >> > > case
> > > >> > > > > from
> > > >> > > > > > > > Pere
> > > >> > > > > > > > > and Enrique (thanks guys!) [1] which uses a load of
> 30
> > > >> > threads
> > > >> > > to
> > > >> > > > > > > > >
> > > >> > > > > > > > > 1) start a new process instance (POST)
> > > >> > > > > > > > > 2) retrieve tasks for a user (GET)
> > > >> > > > > > > > > 3) fetches task details (GET)
> > > >> > > > > > > > > 4) complete a task (POST)
> > > >> > > > > > > > > 5) execute a query on data-audit
> > > >> > > > > > > > >
> > > >> > > > > > > > > With this test setup, I noticed that the performance
> > for
> > > >> the
> > > >> > > POST
> > > >> > > > > > > > > requests, in particular the one to start a new
> process
> > > >> > > instance,
> > > >> > > > > > > degrades
> > > >> > > > > > > > > over time - see graph [2]. If I run the same test
> > > without
> > > >> > > > > data-index,
> > > >> > > > > > > > then
> > > >> > > > > > > > > there is no such performance degradation [3]. You
> can
> > > >> find a
> > > >> > > > thread
> > > >> > > > > > > dump
> > > >> > > > > > > > > captured a few minutes into the first test here [4]
> > that
> > > >> > might
> > > >> > > > help
> > > >> > > > > > to
> > > >> > > > > > > > see
> > > >> > > > > > > > > some of the contention points.
> > > >> > > > > > > > >
> > > >> > > > > > > > > I'd appreciate if you could take a look and see if
> > there
> > > >> is
> > > >> > > > > something
> > > >> > > > > > > > that
> > > >> > > > > > > > > can be further improved based on your previous work.
> > If
> > > >> you
> > > >> > > need
> > > >> > > > > any
> > > >> > > > > > > > > additional data, let me know, but otherwise it is
> > > >> > > straightforward
> > > >> > > > > to
> > > >> > > > > > > run
> > > >> > > > > > > > > the jmeter test as well.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Thanks,
> > > >> > > > > > > > > Martin
> > > >> > > > > > > > >
> > > >> > > > > > > > > [1]
> > > >> https://github.com/pefernan/job-service-refactor-test/
> > > >> > > > > > > > > [2]
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > >> > > > > > > > > [3]
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > >> > > > > > > > > [4]
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > >> > > > > > > > >
> > > >> > > > > > > > > ________________________________________
> > > >> > > > > > > > > From: Francisco Javier Tirado Sarti <
> > > ftirados@redhat.com>
> > > >> > > > > > > > > Sent: Wednesday, January 17, 2024 9:13 AM
> > > >> > > > > > > > > To: dev@kie.apache.org
> > > >> > > > > > > > > Cc: Pere Fernandez Perez
> > > >> > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance
> > issues
> > > >> with
> > > >> > > > > > data-index
> > > >> > > > > > > > > persistence addon
> > > >> > > > > > > > >
> > > >> > > > > > > > > Hi Alex,
> > > >> > > > > > > > > I did not take times (which depends on a number of
> > > >> variables
> > > >> > > that
> > > >> > > > > > > > > drastically change between environments), but verify
> > > that
> > > >> the
> > > >> > > > > number
> > > >> > > > > > of
> > > >> > > > > > > > > updates has been reduced drastically without losing
> > > >> > > > functionality,
> > > >> > > > > > > which
> > > >> > > > > > > > is
> > > >> > > > > > > > > objectively a good thing. If before the change, for
> > > every
> > > >> > node
> > > >> > > > > > > executed,
> > > >> > > > > > > > we
> > > >> > > > > > > > > have an update for every node previously executed,
> so
> > > if a
> > > >> > > > process
> > > >> > > > > > have
> > > >> > > > > > > > 50
> > > >> > > > > > > > > nodes to execute, we were performing nearly 50*51/2
> > > >> updates,
> > > >> > > > which
> > > >> > > > > > > gives
> > > >> > > > > > > > us
> > > >> > > > > > > > > a total of  1275 updates, now we have just one for
> > every
> > > >> node
> > > >> > > > being
> > > >> > > > > > > > > executed, implying a total of 50 updates.
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli <
> > > >> > > alex@porcelli.me>
> > > >> > > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > > Francisco,
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > I noticed that your PR has been merged, but I was
> > > >> expecting
> > > >> > > (at
> > > >> > > > > > least
> > > >> > > > > > > > > > was my understanding from this thread) that before
> > > >> merging
> > > >> > > some
> > > >> > > > > > > > > > benchmark data would be shared in advance - to
> > assess
> > > >> the
> > > >> > > > > > > cost/benefit
> > > >> > > > > > > > > > of such a decent size change.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Do you have any information to share?
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM Francisco Javier
> > > Tirado
> > > >> > Sarti
> > > >> > > > > > > > > > <ft...@redhat.com> wrote:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Yes, as intended, now we have one select and one
> > > >> > > > insert/update
> > > >> > > > > > per
> > > >> > > > > > > > node
> > > >> > > > > > > > > > > event.
> > > >> > > > > > > > > > > I moved the PR as ready for review and give
> @Pere
> > > >> > Fernandez
> > > >> > > > > Perez
> > > >> > > > > > > > > > > <pe...@redhat.com> permission to the branch
> so
> > > he
> > > >> can
> > > >> > > > edit
> > > >> > > > > it
> > > >> > > > > > > in
> > > >> > > > > > > > > the
> > > >> > > > > > > > > > > next two weeks (Ill be on PTO)  if desired,
> before
> > > >> > merging.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex Porcelli <
> > > >> > > > > alex@porcelli.me>
> > > >> > > > > > > > > wrote:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > > Cool, thank you Francisco!
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > Did you manage to get some preliminary data
> > about
> > > >> > > > > improvements?
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM Francisco
> > Javier
> > > >> > Tirado
> > > >> > > > > Sarti
> > > >> > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Yes, after some delay because of quarkus 3
> > > >> migration.
> > > >> > > Im
> > > >> > > > > > > refining
> > > >> > > > > > > > > > this
> > > >> > > > > > > > > > > > > draft PR
> > > >> > > > > > > > > > > > >
> > > >> > > > > >
> > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex
> Porcelli
> > <
> > > >> > > > > > > alex@porcelli.me>
> > > >> > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Any update or new findings on this topic?
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM Francisco
> > > Javier
> > > >> > > Tirado
> > > >> > > > > > Sarti
> > > >> > > > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Hi Alex,
> > > >> > > > > > > > > > > > > > > After considering different options to
> > > improve
> > > >> > > > > > performance,
> > > >> > > > > > > > we
> > > >> > > > > > > > > > feel
> > > >> > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > it
> > > >> > > > > > > > > > > > > > > is time to "partially" move away from
> the
> > > >> current
> > > >> > > Map
> > > >> > > > > > style
> > > >> > > > > > > > > > > > interface (
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > >> > > > > > > > > > > > > > )
> > > >> > > > > > > > > > > > > > > which was shared with Trusty, to one
> more
> > > >> > suitable
> > > >> > > > for
> > > >> > > > > > > usage
> > > >> > > > > > > > > > with a
> > > >> > > > > > > > > > > > > > > relational DB like postgresql (but still
> > > >> > compatible
> > > >> > > > > with
> > > >> > > > > > > big
> > > >> > > > > > > > > > table
> > > >> > > > > > > > > > > > dbs).
> > > >> > > > > > > > > > > > > > > The idea will be to replace generic
> > Storage
> > > >> > > interface
> > > >> > > > > by
> > > >> > > > > > > four
> > > >> > > > > > > > > > > > specific
> > > >> > > > > > > > > > > > > > > interfaces (which will inherit from a
> > common
> > > >> one
> > > >> > > that
> > > >> > > > > > keeps
> > > >> > > > > > > > the
> > > >> > > > > > > > > > query
> > > >> > > > > > > > > > > > > > part
> > > >> > > > > > > > > > > > > > > at is it. with get and query methods),
> > that
> > > >> will
> > > >> > > > > include
> > > >> > > > > > > the
> > > >> > > > > > > > > > required
> > > >> > > > > > > > > > > > > > > modification operations for the four
> > > DataIndex
> > > >> > > > > "domains":
> > > >> > > > > > > > > > > > > > processinstance,
> > > >> > > > > > > > > > > > > > > usertask, processdefinitions and jobs.
> > Those
> > > >> > > > interfaces
> > > >> > > > > > > will
> > > >> > > > > > > > > > define
> > > >> > > > > > > > > > > > > > methods
> > > >> > > > > > > > > > > > > > > like addNode, addVariable, updateTask,
> > > >> > > > > addAttachment.....
> > > >> > > > > > > > that
> > > >> > > > > > > > > > will
> > > >> > > > > > > > > > > > allow
> > > >> > > > > > > > > > > > > > > the persistent layer implementation  to
> > just
> > > >> > update
> > > >> > > > the
> > > >> > > > > > > > needed
> > > >> > > > > > > > > > info
> > > >> > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > DB  (for example, for addNode in
> Postgres,
> > > >> just
> > > >> > > > insert
> > > >> > > > > a
> > > >> > > > > > > row
> > > >> > > > > > > > > into
> > > >> > > > > > > > > > > > nodes
> > > >> > > > > > > > > > > > > > > table, for addNode in Mongo, basically
> the
> > > >> same
> > > >> > > > atomic
> > > >> > > > > > > upsert
> > > >> > > > > > > > > > > > operation
> > > >> > > > > > > > > > > > > > > that is currently done). Therefore, we
> > > >> increase
> > > >> > > > > > performance
> > > >> > > > > > > > for
> > > >> > > > > > > > > > > > Postgres
> > > >> > > > > > > > > > > > > > > and keep the current one for Mongo. The
> > > >> current
> > > >> > DB
> > > >> > > > > > schemas
> > > >> > > > > > > > > won't
> > > >> > > > > > > > > > be
> > > >> > > > > > > > > > > > > > > touched.
> > > >> > > > > > > > > > > > > > > Since the code change is large, I do not
> > > think
> > > >> > I'll
> > > >> > > > be
> > > >> > > > > > able
> > > >> > > > > > > > to
> > > >> > > > > > > > > > have
> > > >> > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > PR
> > > >> > > > > > > > > > > > > > > ready till next week.
> > > >> > > > > > > > > > > > > > > But before starting, please let me know
> if
> > > >> that
> > > >> > > > > approach
> > > >> > > > > > is
> > > >> > > > > > > > > fine
> > > >> > > > > > > > > > for
> > > >> > > > > > > > > > > > you.
> > > >> > > > > > > > > > > > > > > Best regards.
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 6:55 PM Alex
> > > Porcelli
> > > >> <
> > > >> > > > > > > > > alex@porcelli.me>
> > > >> > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > Thank you Francisco to getting deeper
> on
> > > >> this…
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > Looking forward to see the results of
> > your
> > > >> > > > suggested
> > > >> > > > > > > > > > improvements.
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 9:40 AM
> > Francisco
> > > >> > Javier
> > > >> > > > > Tirado
> > > >> > > > > > > > > Sarti <
> > > >> > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > I forgot to attach the queries
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 3:04 PM
> > > Francisco
> > > >> > > Javier
> > > >> > > > > > Tirado
> > > >> > > > > > > > > > Sarti <
> > > >> > > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > >> Hi,
> > > >> > > > > > > > > > > > > > > > >> A brief update on this topic.
> > > >> > > > > > > > > > > > > > > > >> After doing a simple test with
> > example
> > > >> > > > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > >> > > > > > > > > > > > > > > > ,
> > > >> > > > > > > > > > > > > > > > >> the number of updates over Nodes
> > table
> > > is
> > > >> > n*n,
> > > >> > > > so
> > > >> > > > > we
> > > >> > > > > > > > > manage
> > > >> > > > > > > > > > to
> > > >> > > > > > > > > > > > > > obtain a
> > > >> > > > > > > > > > > > > > > > >> perfect quadratic performance
> > > >> degradation.
> > > >> > The
> > > >> > > > > > problem
> > > >> > > > > > > > is
> > > >> > > > > > > > > > worse
> > > >> > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > case
> > > >> > > > > > > > > > > > > > > > >> of Serverless Workflow than in BPMN
> > > >> because
> > > >> > we
> > > >> > > > the
> > > >> > > > > > > > number
> > > >> > > > > > > > > of
> > > >> > > > > > > > > > > > nodes
> > > >> > > > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > > > >> greater than the number of states.
> In
> > > >> that
> > > >> > > > > example N
> > > >> > > > > > > is
> > > >> > > > > > > > > 16,
> > > >> > > > > > > > > > but
> > > >> > > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > a
> > > >> > > > > > > > > > > > > > > > more
> > > >> > > > > > > > > > > > > > > > >> complex workflow it would be
> > certainly
> > > >> > large.
> > > >> > > > > > > > > > > > > > > > >> I think that this is more related
> to
> > > how
> > > >> we
> > > >> > > are
> > > >> > > > > > > handling
> > > >> > > > > > > > > > JPA in
> > > >> > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > code,
> > > >> > > > > > > > > > > > > > > > >> in particular the mapping from
> model
> > to
> > > >> > entity
> > > >> > > > > > > > (basically
> > > >> > > > > > > > > > JPA is
> > > >> > > > > > > > > > > > > > blind
> > > >> > > > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > >> has to update all nodes for every
> > write
> > > >> > > because
> > > >> > > > it
> > > >> > > > > > > > > believes
> > > >> > > > > > > > > > the
> > > >> > > > > > > > > > > > > > node has
> > > >> > > > > > > > > > > > > > > > >> been updated, although it is not)
> > than
> > > an
> > > >> > > issue
> > > >> > > > in
> > > >> > > > > > the
> > > >> > > > > > > > > table
> > > >> > > > > > > > > > > > > > definition.
> > > >> > > > > > > > > > > > > > > > >> In fact, when using JPA, separating
> > the
> > > >> > server
> > > >> > > > > model
> > > >> > > > > > > > from
> > > >> > > > > > > > > > the
> > > >> > > > > > > > > > > > JPA
> > > >> > > > > > > > > > > > > > > > entity is
> > > >> > > > > > > > > > > > > > > > >> not a good idea, especially if the
> > > entity
> > > >> > > > contains
> > > >> > > > > > > > > > collections.
> > > >> > > > > > > > > > > > I
> > > >> > > > > > > > > > > > > > will
> > > >> > > > > > > > > > > > > > > > try
> > > >> > > > > > > > > > > > > > > > >> to change that without breaking
> > > anything.
> > > >> > > > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > > > >> On Wed, Nov 22, 2023 at 12:10 PM
> > > Enrique
> > > >> > > > Gonzalez
> > > >> > > > > > > > > Martinez <
> > > >> > > > > > > > > > > > > > > > >> egonzalez@apache.org> wrote:
> > > >> > > > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > > > >>> After the events split you now
> will
> > > >> need to
> > > >> > > > > create
> > > >> > > > > > a
> > > >> > > > > > > > node
> > > >> > > > > > > > > > > > instance
> > > >> > > > > > > > > > > > > > > > >>> model instance of making
> independent
> > > >> from
> > > >> > the
> > > >> > > > > > process
> > > >> > > > > > > > > > instance.
> > > >> > > > > > > > > > > > > > > > >>> That should do the trick.
> > > >> > > > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > > > >>> Regarding deleting/inserting it
> was
> > > >> fixed
> > > >> > at
> > > >> > > > some
> > > >> > > > > > > > point.
> > > >> > > > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > > > >>> El mar, 21 nov 2023 a las 20:22,
> > > >> Francisco
> > > >> > > > Javier
> > > >> > > > > > > > Tirado
> > > >> > > > > > > > > > Sarti
> > > >> > > > > > > > > > > > > > > > >>> (<ft...@redhat.com>) escribió:
> > > >> > > > > > > > > > > > > > > > >>> >
> > > >> > > > > > > > > > > > > > > > >>> > Hi Martin,
> > > >> > > > > > > > > > > > > > > > >>> > I have a task to review
> > performance
> > > of
> > > >> > > > > > > > > > > > > > > > >>> >
> > > >> > > > > > > > > > > > > > > > >>> >
> ProcessInstanceNodeDataEventMerger
> > > >> > > > > > > > > > > > > > > > >>> > My idea is to reduce the number
> of
> > > >> delete
> > > >> > > > > inserts
> > > >> > > > > > > > when
> > > >> > > > > > > > > > > > processing
> > > >> > > > > > > > > > > > > > > > >>> events
> > > >> > > > > > > > > > > > > > > > >>> > and try to do it incremental.
> > > >> > > > > > > > > > > > > > > > >>> > That should improve performance.
> > > >> > > > > > > > > > > > > > > > >>> > PS:
> > > >> > > > > > > > > > > > > > > > >>> > I was planning to send an e-mail
> > > >> tomorrow
> > > >> > > > > > > announcing
> > > >> > > > > > > > > > that in
> > > >> > > > > > > > > > > > > > case you
> > > >> > > > > > > > > > > > > > > > >>> were
> > > >> > > > > > > > > > > > > > > > >>> > already working on a fix for
> > that. I
> > > >> > assume
> > > >> > > > you
> > > >> > > > > > are
> > > >> > > > > > > > not
> > > >> > > > > > > > > > and I
> > > >> > > > > > > > > > > > > > would
> > > >> > > > > > > > > > > > > > > > be
> > > >> > > > > > > > > > > > > > > > >>> > sending a PR soon.
> > > >> > > > > > > > > > > > > > > > >>> >
> > > >> > > > > > > > > > > > > > > > >>> > On Tue, Nov 21, 2023 at 6:09 PM
> > > Martin
> > > >> > > Weiler
> > > >> > > > > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > >> > > > > > > > > > > > > > > > >>> >
> > > >> > > > > > > > > > > > > > > > >>> > wrote:
> > > >> > > > > > > > > > > > > > > > >>> >
> > > >> > > > > > > > > > > > > > > > >>> > > I looked into the new examples
> > > using
> > > >> > > > > data-index
> > > >> > > > > > > > > > persistence
> > > >> > > > > > > > > > > > > > addon -
> > > >> > > > > > > > > > > > > > > > >>> Neus'
> > > >> > > > > > > > > > > > > > > > >>> > > PR#1813 [1] for serverless and
> > > >> Pere's
> > > >> > > > branch
> > > >> > > > > > [2]
> > > >> > > > > > > > for
> > > >> > > > > > > > > > > > workflow
> > > >> > > > > > > > > > > > > > > > (great
> > > >> > > > > > > > > > > > > > > > >>> job
> > > >> > > > > > > > > > > > > > > > >>> > > both!) - and they work without
> > > >> issues
> > > >> > > using
> > > >> > > > > > > single
> > > >> > > > > > > > > > > > requests.
> > > >> > > > > > > > > > > > > > > > >>> However, under
> > > >> > > > > > > > > > > > > > > > >>> > > some load (I used 'ab' for
> > testing
> > > >> > with a
> > > >> > > > > light
> > > >> > > > > > > > > > > > concurrency of
> > > >> > > > > > > > > > > > > > 10
> > > >> > > > > > > > > > > > > > > > >>> parallel
> > > >> > > > > > > > > > > > > > > > >>> > > requests) I ran into the
> > following
> > > >> > > > problems:
> > > >> > > > > > > > > > > > > > > > >>> > >
> > > >> > > > > > > > > > > > > > > > >>> > > (1) Large number of
> > insert/delete
> > > >> calls
> > > >> > > > (eg.
> > > >> > > > > > for
> > > >> > > > > > > > > tables
> > > >> > > > > > > > > > > > such as
> > > >> > > > > > > > > > > > > > > > >>> nodes,
> > > >> > > > > > > > > > > > > > > > >>> > > definitions, etc)
> > > >> > > > > > > > > > > > > > > > >>> > >
> > > >> > > > > > > > > > > > > > > > >>> > > (2) Hibernate
> > > >> OptimisticLockExceptions
> > > >> > /
> > > >> > > > > > > > > > > > StaleStateExceptions
> > > >> > > > > > > > > > > > > > > > >>> > >
> > > >> > > > > > > > > > > > > > > > >>> > > (3) DB deadlocks
> > > >> > > > > > > > > > > > > > > > >>> > >
> > > >> > > > > > > > > > > > > > > > >>> > > (4) Error responses, slow
> > response
> > > >> > times
> > > >> > > > > > > > > > > > > > > > >>> > >
> > > >> > > > > > > > > > > > > > > > >>> > > The reason I am reaching out
> > with
> > > >> this
> > > >> > > > topic
> > > >> > > > > > here
> > > >> > > > > > > > is
> > > >> > > > > > > > > to
> > > >> > > > > > > > > > > > find
> > > >> > > > > > > > > > > > > > out if
> > > >> > > > > > > > > > > > > > > > >>> we are
> > > >> > > > > > > > > > > > > > > > >>> > > aware of this issue, and if
> > > someone
> > > >> is
> > > >> > > > > already
> > > >> > > > > > > > > looking
> > > >> > > > > > > > > > > > into or
> > > >> > > > > > > > > > > > > > > > being
> > > >> > > > > > > > > > > > > > > > >>> > > assigned to it?
> > > >> > > > > > > > > > > > > > > > >>> > >
> > > >> > > > > > > > > > > > > > > > >>> > > Thanks,
> > > >> > > > > > > > > > > > > > > > >>> > > Martin
> > > >> > > > > > > > > > > > > > > > >>> > >
> > > >> > > > > > > > > > > > > > > > >>> > > [1]
> > > >> > > > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > >
> > > >> > > > > > >
> > > >> > https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > >> > > > > > > > > > > > > > > > >>> > > [2]
> > > >> > > > > > > > > > > > > > > > >>> > >
> > > >> > > > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > >> > > > > > > > > > > > > > > > >>> > >
> > > >> > > > > > > > > > > > > > > > >>> > >
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > >
> > > >> > > >
> > > >>
> ---------------------------------------------------------------------
> > > >> > > > > > > > > > > > > > > > >>> > > To unsubscribe, e-mail:
> > > >> > > > > > > > > dev-unsubscribe@kie.apache.org
> > > >> > > > > > > > > > > > > > > > >>> > > For additional commands,
> e-mail:
> > > >> > > > > > > > > > dev-help@kie.apache.org
> > > >> > > > > > > > > > > > > > > > >>> > >
> > > >> > > > > > > > > > > > > > > > >>> > >
> > > >> > > > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > >
> > > >> >
> > ---------------------------------------------------------------------
> > > >> > > > > > > > > > > > > > > > >>> To unsubscribe, e-mail:
> > > >> > > > > > > dev-unsubscribe@kie.apache.org
> > > >> > > > > > > > > > > > > > > > >>> For additional commands, e-mail:
> > > >> > > > > > > > dev-help@kie.apache.org
> > > >> > > > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > >
> > > >> > > >
> > > >>
> ---------------------------------------------------------------------
> > > >> > > > > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > >> > > > > > dev-unsubscribe@kie.apache.org
> > > >> > > > > > > > > > > > > > > > > For additional commands, e-mail:
> > > >> > > > > > > dev-help@kie.apache.org
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > >
> > > >> >
> > ---------------------------------------------------------------------
> > > >> > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > >> > > dev-unsubscribe@kie.apache.org
> > > >> > > > > > > > > > > > > > For additional commands, e-mail:
> > > >> > > > dev-help@kie.apache.org
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > >
> > > >> > > >
> > > >>
> ---------------------------------------------------------------------
> > > >> > > > > > > > > > > > To unsubscribe, e-mail:
> > > >> dev-unsubscribe@kie.apache.org
> > > >> > > > > > > > > > > > For additional commands, e-mail:
> > > >> > dev-help@kie.apache.org
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > >
> > > >> >
> > ---------------------------------------------------------------------
> > > >> > > > > > > > > > To unsubscribe, e-mail:
> > > dev-unsubscribe@kie.apache.org
> > > >> > > > > > > > > > For additional commands, e-mail:
> > > >> dev-help@kie.apache.org
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > >
> > > >>
> ---------------------------------------------------------------------
> > > >> > > > > > > > > To unsubscribe, e-mail:
> > dev-unsubscribe@kie.apache.org
> > > >> > > > > > > > > For additional commands, e-mail:
> > > dev-help@kie.apache.org
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >>
> ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > >> For additional commands, e-mail: dev-help@kie.apache.org
> > > >>
> > > >>
> > >
> >
>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Francisco Javier Tirado Sarti <ft...@redhat.com>]

Hi Alex,
There has been some confusion about the purpose of DataIndex. To be honest
I believe they were already sorted out, but your e-mail makes me think that
is not the case ;). I let Kris to clarify that with you. My view is that
data-index is a way to query recently closed and active processes (the key
here is the definition of recently, which in my opinion should be
configurable)
But, besides that discussion and being pragmatic, keeping finishing process
instances "for a while" in DataIndex was the only way for users to query
the result of straight through processes. That's a function that cannot be
removed right now

On Mon, Feb 19, 2024 at 1:33 PM Alex Porcelli <po...@apache.org> wrote:

> if data index was supposed to provide snapshot view of the process
> instance… why do we keep it after the process instance is finished?
>
>
> On Mon, Feb 19, 2024 at 7:12 AM Francisco Javier Tirado Sarti <
> ftirados@redhat.com> wrote:
>
> > Hi Martin.
> > After taking a deeper look at this, I realize that the behaviour is the
> > expected one.
> > Runtimes DB does not track the completed process instance (that's what
> the
> > JDBCProcessInstances warn is telling us), but DataIndex, as expected, is
> > tracking it in processes and nodes table. And yes it will grow over time.
> > What we need is some configurable purge mechanism for DataIndex, so it
> > eventually removes older completed process instances.
> >
> > On Tue, Feb 13, 2024 at 12:59 PM Francisco Javier Tirado Sarti <
> > ftirados@redhat.com> wrote:
> >
> > > Hi Martin,
> > > Good catch!. Looks like the skipping performed for process instances is
> > > not applied to node instances. Something we definitely need to review
> on
> > > the runtimes side.
> > >
> > > On Mon, Feb 12, 2024 at 11:59 PM Martin Weiler <mweiler@ibm.com.invalid
> >
> > > wrote:
> > >
> > >> On a somewhat related note, testing a simple workflow (start -> script
> > >> node -> end), I see the following messages in the logs:
> > >> 2024-02-12 22:49:50,493 28758dde544c WARN
> > >> [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> > >> (executor-thread-3) Skipping create of process instance id:
> > >> 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2
> > >>
> > >> So far, so good. And I'd expect to see no trace of this process in the
> > >> database if I don't have data audit enabled.
> > >>
> > >> However, the 'processes' table contains a row with state=2, with
> related
> > >> entries in the 'nodes' table. In a load test, I see these tables grow
> > >> significantly over time. Am I missing something to have these entries
> > >> cleaned up automatically?
> > >>
> > >> ________________________________________
> > >> From: Martin Weiler <mw...@ibm.com.INVALID>
> > >> Sent: Monday, February 12, 2024 3:40 PM
> > >> To: dev@kie.apache.org
> > >> Subject: [EXTERNAL] RE: [DISCUSSION] Performance issues with
> data-index
> > >> persistence addon
> > >>
> > >> Thanks everyone for your input. Based on this discussion, I opened the
> > >> following PR:
> > >> https://github.com/apache/incubator-kie-kogito-apps/pull/1985
> > >>
> > >> With this change, the performance seems to be stable over time:
> > >>
> > >>
> >
> https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
> > >>
> > >> Martin
> > >>
> > >> ________________________________________
> > >> From: Gonzalo Muñoz <gm...@apache.org>
> > >> Sent: Friday, February 9, 2024 9:42 AM
> > >> To: dev@kie.apache.org
> > >> Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> data-index
> > >> persistence addon
> > >>
> > >> Great work Francisco,
> > >> Martin, take a look at this link with some related tips (in case you
> > find
> > >> it useful):
> > >> https://www.cybertec-postgresql.com/en/index-your-foreign-key/
> > >>
> > >> El vie, 9 feb 2024 a las 17:20, Francisco Javier Tirado Sarti (<
> > >> ftirados@redhat.com>) escribió:
> > >>
> > >> > For the moment being, we will keep JPA till we exhaust all
> > >> possibilities,
> > >> > let's call switching from jpa to jdbc our hidden plan B ;)
> > >> > I already told Martin, but in order everyone to know, just after
> > writing
> > >> > the previous email, I thought "what if Postgres is not automatically
> > >> > indexing foreign keys like mysql?" and, eureka
> > >> > Postgres doc
> > >> > https://www.postgresql.org/docs/current/ddl-constraints.html
> > >> > Mysql doc
> > >> > https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> > >> > These are the relevant excerpt
> > >> >
> > >> > *Postgresql*
> > >> > *A foreign key must reference columns that either are a primary key
> or
> > >> form
> > >> > a unique constraint, or are columns from a non-partial unique index.
> > >> This
> > >> > means that the referenced columns always have an index to allow
> > >> efficient
> > >> > lookups on whether a referencing row has a match. Since a DELETE of
> a
> > >> row
> > >> > from the referenced table or an UPDATE of a referenced column will
> > >> require
> > >> > a scan of the referencing table for rows matching the old value, it
> is
> > >> > often a good idea to index the referencing columns too. Because this
> > is
> > >> not
> > >> > always needed, and there are many choices available on how to index,
> > the
> > >> > declaration of a foreign key constraint does not automatically
> create
> > an
> > >> > index on the referencing columns.*
> > >> > *Mysql*
> > >> > *MySQL requires that foreign key columns be indexed; if you create a
> > >> table
> > >> > with a foreign key constraint but no index on a given column, an
> index
> > >> is
> > >> > created. *
> > >> >
> > >> > So I asked Martin to especially create an index for
> > process_instance_id
> > >> > column on nodes table
> > >> > I think that will fix the problem detected on the thread dump.
> > >> > The simpler process test to verify queries are fine still stands,
> > >> though ;)
> > >> >
> > >> >
> > >> > On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <tz...@apache.org>
> > >> wrote:
> > >> >
> > >> > > I always preferred pure JDBC over Hibernate myself, just for the
> > sake
> > >> of
> > >> > > control of what is happening :) So I would not -1 that myself.
> > >> > >
> > >> > > Tibor
> > >> > >
> > >> > > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado Sarti <
> > >> > > ftirados@redhat.com>
> > >> > > napísal(a):
> > >> > >
> > >> > > > Hi,
> > >> > > > Usually I do not want to talk about work in progress because
> > >> > preliminary
> > >> > > > conclusions are pretty volatile but, well, there are a couple of
> > >> things
> > >> > > > that can be concluded from the really valuable information that
> > >> Martin
> > >> > > > provided.
> > >> > > > 1) In order to be able to determine if the number of statements
> is
> > >> > larger
> > >> > > > than expected, I asked Martin to test with a simpler process
> > >> > definition.
> > >> > > > One with just three nodes: start, script and end. The script one
> > >> should
> > >> > > > change just one variable. This way we can analyze if the number
> of
> > >> > > queries
> > >> > > > is the expected one. From the single log (audit was activated
> > them)
> > >> my
> > >> > > > conclusion is that the number of insert/updates over processes
> and
> > >> > nodes
> > >> > > > (there a lot over task, that I will prefer to skip for now, baby
> > >> steps)
> > >> > > is
> > >> > > > the expected one.
> > >> > > > 2) Analysing the thread dump, we see around 15 threads executing
> > >> this
> > >> > > line
> > >> > > > at
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > >> > > > so its pretty clear the code to be optimized ;). I'm evaluating
> > >> > > > possibilities within JPA/Hibernate, but I'm starting to think
> that
> > >> it
> > >> > > might
> > >> > > > be better to switch to JDBC and skip hibernate. Our lives will
> be
> > >> > > simpler,
> > >> > > > especially with a schema relatively simple like ours (that will
> be
> > >> my
> > >> > > > recommendation if I was an external consultant)
> > >> > > >
> > >> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <
> tzimanyi@apache.org
> > >
> > >> > > wrote:
> > >> > > >
> > >> > > > > Hi,
> > >> > > > >
> > >> > > > > this will be a bit off-topic. However as far as performance, I
> > >> think
> > >> > we
> > >> > > > > should think about that we have string primary keys (IDs). I
> > would
> > >> > > expect
> > >> > > > > the database systems are much better with indexing numeric
> keys
> > >> than
> > >> > > > > strings. I remember from the past, when I was working with
> DBs,
> > >> that
> > >> > > > using
> > >> > > > > strings as keys or indexes was a discouraged practice.
> > >> > > > >
> > >> > > > > Best regards,
> > >> > > > > Tibor
> > >> > > > >
> > >> > > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler
> <mweiler@ibm.com.invalid
> > >
> > >> > > > > napísal(a):
> > >> > > > >
> > >> > > > > > I changed the test to use MongoDB [1] and I don't see a
> > >> performance
> > >> > > > > > degradation with this setup [2].
> > >> > > > > >
> > >> > > > > > Please keep us posted of your findings. Thanks!
> > >> > > > > >
> > >> > > > > > Martin
> > >> > > > > >
> > >> > > > > > [1]
> > >> > > > >
> > >> >
> > https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > >> > > > > > [2]
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > >> > > > > >
> > >> > > > > > ________________________________________
> > >> > > > > > From: Francisco Javier Tirado Sarti <ft...@redhat.com>
> > >> > > > > > Sent: Wednesday, February 7, 2024 11:40 AM
> > >> > > > > > To: dev@kie.apache.org
> > >> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > >> > > data-index
> > >> > > > > > persistence addon
> > >> > > > > >
> > >> > > > > > yes, it can be index degradation because of size, but I
> > believe
> > >> (I
> > >> > > > might
> > >> > > > > be
> > >> > > > > > wrong) the db is too small (yet) for that.
> > >> > > > > > But, eventually, Postgres, when the DB is huge enough,
> > >> unavoidably
> > >> > > will
> > >> > > > > > behave like the graphic that Martin sent.
> > >> > > > > > Since I believe we are not huge enough (yet), lets rule out
> > >> another
> > >> > > > issue
> > >> > > > > > by analysing the sql logs (I requested those to Martin
> offline
> > >> and
> > >> > he
> > >> > > > is
> > >> > > > > > going to kindly collect them).
> > >> > > > > > Also Im curious to know if Mongo behave in the same way.
> > >> > > > > >
> > >> > > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez Martinez <
> > >> > > > > > egonzalez@apache.org> wrote:
> > >> > > > > >
> > >> > > > > > > Hi Francisco,
> > >> > > > > > > I would highly recommend to check indexes and how the
> > updates
> > >> > work
> > >> > > in
> > >> > > > > > data
> > >> > > > > > > index to avoid full scan table and lock the full table.
> Some
> > >> db
> > >> > are
> > >> > > > > very
> > >> > > > > > > sensitive to that.
> > >> > > > > > >
> > >> > > > > > > El mié, 7 feb 2024, 18:41, Francisco Javier Tirado Sarti <
> > >> > > > > > > ftirados@redhat.com> escribió:
> > >> > > > > > >
> > >> > > > > > > > Hi Martin,
> > >> > > > > > > > While I analyze the data, let me ask you if it is
> possible
> > >> to
> > >> > > > perform
> > >> > > > > > > > another check (similar in a way to disabling data-index
> > like
> > >> > you
> > >> > > > do)
> > >> > > > > > Can
> > >> > > > > > > > you switch to MongoDB persistence and check if the same
> > >> > > degradation
> > >> > > > > > that
> > >> > > > > > > is
> > >> > > > > > > > there for postgres remains?
> > >> > > > > > > > I do not know if this is feasible but will certainly
> > >> indicate
> > >> > the
> > >> > > > > > problem
> > >> > > > > > > > is on the postgres storage layer and I do not have a
> clear
> > >> > > > prediction
> > >> > > > > > of
> > >> > > > > > > > what we will see when doing this switch.
> > >> > > > > > > >
> > >> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler
> > >> > > > <mweiler@ibm.com.invalid
> > >> > > > > >
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > Hi Francisco,
> > >> > > > > > > > >
> > >> > > > > > > > > thanks for your work on this important topic!
> > >> > > > > > > > >
> > >> > > > > > > > > I would like to share some test results here, which
> > might
> > >> > help
> > >> > > to
> > >> > > > > > > improve
> > >> > > > > > > > > the codebase even further. I am using the jmeter based
> > >> test
> > >> > > case
> > >> > > > > from
> > >> > > > > > > > Pere
> > >> > > > > > > > > and Enrique (thanks guys!) [1] which uses a load of 30
> > >> > threads
> > >> > > to
> > >> > > > > > > > >
> > >> > > > > > > > > 1) start a new process instance (POST)
> > >> > > > > > > > > 2) retrieve tasks for a user (GET)
> > >> > > > > > > > > 3) fetches task details (GET)
> > >> > > > > > > > > 4) complete a task (POST)
> > >> > > > > > > > > 5) execute a query on data-audit
> > >> > > > > > > > >
> > >> > > > > > > > > With this test setup, I noticed that the performance
> for
> > >> the
> > >> > > POST
> > >> > > > > > > > > requests, in particular the one to start a new process
> > >> > > instance,
> > >> > > > > > > degrades
> > >> > > > > > > > > over time - see graph [2]. If I run the same test
> > without
> > >> > > > > data-index,
> > >> > > > > > > > then
> > >> > > > > > > > > there is no such performance degradation [3]. You can
> > >> find a
> > >> > > > thread
> > >> > > > > > > dump
> > >> > > > > > > > > captured a few minutes into the first test here [4]
> that
> > >> > might
> > >> > > > help
> > >> > > > > > to
> > >> > > > > > > > see
> > >> > > > > > > > > some of the contention points.
> > >> > > > > > > > >
> > >> > > > > > > > > I'd appreciate if you could take a look and see if
> there
> > >> is
> > >> > > > > something
> > >> > > > > > > > that
> > >> > > > > > > > > can be further improved based on your previous work.
> If
> > >> you
> > >> > > need
> > >> > > > > any
> > >> > > > > > > > > additional data, let me know, but otherwise it is
> > >> > > straightforward
> > >> > > > > to
> > >> > > > > > > run
> > >> > > > > > > > > the jmeter test as well.
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks,
> > >> > > > > > > > > Martin
> > >> > > > > > > > >
> > >> > > > > > > > > [1]
> > >> https://github.com/pefernan/job-service-refactor-test/
> > >> > > > > > > > > [2]
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > >> > > > > > > > > [3]
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > >> > > > > > > > > [4]
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > >> > > > > > > > >
> > >> > > > > > > > > ________________________________________
> > >> > > > > > > > > From: Francisco Javier Tirado Sarti <
> > ftirados@redhat.com>
> > >> > > > > > > > > Sent: Wednesday, January 17, 2024 9:13 AM
> > >> > > > > > > > > To: dev@kie.apache.org
> > >> > > > > > > > > Cc: Pere Fernandez Perez
> > >> > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance
> issues
> > >> with
> > >> > > > > > data-index
> > >> > > > > > > > > persistence addon
> > >> > > > > > > > >
> > >> > > > > > > > > Hi Alex,
> > >> > > > > > > > > I did not take times (which depends on a number of
> > >> variables
> > >> > > that
> > >> > > > > > > > > drastically change between environments), but verify
> > that
> > >> the
> > >> > > > > number
> > >> > > > > > of
> > >> > > > > > > > > updates has been reduced drastically without losing
> > >> > > > functionality,
> > >> > > > > > > which
> > >> > > > > > > > is
> > >> > > > > > > > > objectively a good thing. If before the change, for
> > every
> > >> > node
> > >> > > > > > > executed,
> > >> > > > > > > > we
> > >> > > > > > > > > have an update for every node previously executed, so
> > if a
> > >> > > > process
> > >> > > > > > have
> > >> > > > > > > > 50
> > >> > > > > > > > > nodes to execute, we were performing nearly 50*51/2
> > >> updates,
> > >> > > > which
> > >> > > > > > > gives
> > >> > > > > > > > us
> > >> > > > > > > > > a total of  1275 updates, now we have just one for
> every
> > >> node
> > >> > > > being
> > >> > > > > > > > > executed, implying a total of 50 updates.
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli <
> > >> > > alex@porcelli.me>
> > >> > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > Francisco,
> > >> > > > > > > > > >
> > >> > > > > > > > > > I noticed that your PR has been merged, but I was
> > >> expecting
> > >> > > (at
> > >> > > > > > least
> > >> > > > > > > > > > was my understanding from this thread) that before
> > >> merging
> > >> > > some
> > >> > > > > > > > > > benchmark data would be shared in advance - to
> assess
> > >> the
> > >> > > > > > > cost/benefit
> > >> > > > > > > > > > of such a decent size change.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Do you have any information to share?
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM Francisco Javier
> > Tirado
> > >> > Sarti
> > >> > > > > > > > > > <ft...@redhat.com> wrote:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Yes, as intended, now we have one select and one
> > >> > > > insert/update
> > >> > > > > > per
> > >> > > > > > > > node
> > >> > > > > > > > > > > event.
> > >> > > > > > > > > > > I moved the PR as ready for review and give @Pere
> > >> > Fernandez
> > >> > > > > Perez
> > >> > > > > > > > > > > <pe...@redhat.com> permission to the branch so
> > he
> > >> can
> > >> > > > edit
> > >> > > > > it
> > >> > > > > > > in
> > >> > > > > > > > > the
> > >> > > > > > > > > > > next two weeks (Ill be on PTO)  if desired, before
> > >> > merging.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex Porcelli <
> > >> > > > > alex@porcelli.me>
> > >> > > > > > > > > wrote:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > > Cool, thank you Francisco!
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Did you manage to get some preliminary data
> about
> > >> > > > > improvements?
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM Francisco
> Javier
> > >> > Tirado
> > >> > > > > Sarti
> > >> > > > > > > > > > > > <ft...@redhat.com> wrote:
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Yes, after some delay because of quarkus 3
> > >> migration.
> > >> > > Im
> > >> > > > > > > refining
> > >> > > > > > > > > > this
> > >> > > > > > > > > > > > > draft PR
> > >> > > > > > > > > > > > >
> > >> > > > > >
> https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex Porcelli
> <
> > >> > > > > > > alex@porcelli.me>
> > >> > > > > > > > > > wrote:
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Any update or new findings on this topic?
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM Francisco
> > Javier
> > >> > > Tirado
> > >> > > > > > Sarti
> > >> > > > > > > > > > > > > > <ft...@redhat.com> wrote:
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Hi Alex,
> > >> > > > > > > > > > > > > > > After considering different options to
> > improve
> > >> > > > > > performance,
> > >> > > > > > > > we
> > >> > > > > > > > > > feel
> > >> > > > > > > > > > > > that
> > >> > > > > > > > > > > > > > it
> > >> > > > > > > > > > > > > > > is time to "partially" move away from the
> > >> current
> > >> > > Map
> > >> > > > > > style
> > >> > > > > > > > > > > > interface (
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > >> > > > > > > > > > > > > > )
> > >> > > > > > > > > > > > > > > which was shared with Trusty, to one more
> > >> > suitable
> > >> > > > for
> > >> > > > > > > usage
> > >> > > > > > > > > > with a
> > >> > > > > > > > > > > > > > > relational DB like postgresql (but still
> > >> > compatible
> > >> > > > > with
> > >> > > > > > > big
> > >> > > > > > > > > > table
> > >> > > > > > > > > > > > dbs).
> > >> > > > > > > > > > > > > > > The idea will be to replace generic
> Storage
> > >> > > interface
> > >> > > > > by
> > >> > > > > > > four
> > >> > > > > > > > > > > > specific
> > >> > > > > > > > > > > > > > > interfaces (which will inherit from a
> common
> > >> one
> > >> > > that
> > >> > > > > > keeps
> > >> > > > > > > > the
> > >> > > > > > > > > > query
> > >> > > > > > > > > > > > > > part
> > >> > > > > > > > > > > > > > > at is it. with get and query methods),
> that
> > >> will
> > >> > > > > include
> > >> > > > > > > the
> > >> > > > > > > > > > required
> > >> > > > > > > > > > > > > > > modification operations for the four
> > DataIndex
> > >> > > > > "domains":
> > >> > > > > > > > > > > > > > processinstance,
> > >> > > > > > > > > > > > > > > usertask, processdefinitions and jobs.
> Those
> > >> > > > interfaces
> > >> > > > > > > will
> > >> > > > > > > > > > define
> > >> > > > > > > > > > > > > > methods
> > >> > > > > > > > > > > > > > > like addNode, addVariable, updateTask,
> > >> > > > > addAttachment.....
> > >> > > > > > > > that
> > >> > > > > > > > > > will
> > >> > > > > > > > > > > > allow
> > >> > > > > > > > > > > > > > > the persistent layer implementation  to
> just
> > >> > update
> > >> > > > the
> > >> > > > > > > > needed
> > >> > > > > > > > > > info
> > >> > > > > > > > > > > > in
> > >> > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > DB  (for example, for addNode in Postgres,
> > >> just
> > >> > > > insert
> > >> > > > > a
> > >> > > > > > > row
> > >> > > > > > > > > into
> > >> > > > > > > > > > > > nodes
> > >> > > > > > > > > > > > > > > table, for addNode in Mongo, basically the
> > >> same
> > >> > > > atomic
> > >> > > > > > > upsert
> > >> > > > > > > > > > > > operation
> > >> > > > > > > > > > > > > > > that is currently done). Therefore, we
> > >> increase
> > >> > > > > > performance
> > >> > > > > > > > for
> > >> > > > > > > > > > > > Postgres
> > >> > > > > > > > > > > > > > > and keep the current one for Mongo. The
> > >> current
> > >> > DB
> > >> > > > > > schemas
> > >> > > > > > > > > won't
> > >> > > > > > > > > > be
> > >> > > > > > > > > > > > > > > touched.
> > >> > > > > > > > > > > > > > > Since the code change is large, I do not
> > think
> > >> > I'll
> > >> > > > be
> > >> > > > > > able
> > >> > > > > > > > to
> > >> > > > > > > > > > have
> > >> > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > PR
> > >> > > > > > > > > > > > > > > ready till next week.
> > >> > > > > > > > > > > > > > > But before starting, please let me know if
> > >> that
> > >> > > > > approach
> > >> > > > > > is
> > >> > > > > > > > > fine
> > >> > > > > > > > > > for
> > >> > > > > > > > > > > > you.
> > >> > > > > > > > > > > > > > > Best regards.
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 6:55 PM Alex
> > Porcelli
> > >> <
> > >> > > > > > > > > alex@porcelli.me>
> > >> > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > Thank you Francisco to getting deeper on
> > >> this…
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > Looking forward to see the results of
> your
> > >> > > > suggested
> > >> > > > > > > > > > improvements.
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 9:40 AM
> Francisco
> > >> > Javier
> > >> > > > > Tirado
> > >> > > > > > > > > Sarti <
> > >> > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > I forgot to attach the queries
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 3:04 PM
> > Francisco
> > >> > > Javier
> > >> > > > > > Tirado
> > >> > > > > > > > > > Sarti <
> > >> > > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > >> Hi,
> > >> > > > > > > > > > > > > > > > >> A brief update on this topic.
> > >> > > > > > > > > > > > > > > > >> After doing a simple test with
> example
> > >> > > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > >> > > > > > > > > > > > > > > > ,
> > >> > > > > > > > > > > > > > > > >> the number of updates over Nodes
> table
> > is
> > >> > n*n,
> > >> > > > so
> > >> > > > > we
> > >> > > > > > > > > manage
> > >> > > > > > > > > > to
> > >> > > > > > > > > > > > > > obtain a
> > >> > > > > > > > > > > > > > > > >> perfect quadratic performance
> > >> degradation.
> > >> > The
> > >> > > > > > problem
> > >> > > > > > > > is
> > >> > > > > > > > > > worse
> > >> > > > > > > > > > > > in
> > >> > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > case
> > >> > > > > > > > > > > > > > > > >> of Serverless Workflow than in BPMN
> > >> because
> > >> > we
> > >> > > > the
> > >> > > > > > > > number
> > >> > > > > > > > > of
> > >> > > > > > > > > > > > nodes
> > >> > > > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > > > >> greater than the number of states. In
> > >> that
> > >> > > > > example N
> > >> > > > > > > is
> > >> > > > > > > > > 16,
> > >> > > > > > > > > > but
> > >> > > > > > > > > > > > for
> > >> > > > > > > > > > > > > > a
> > >> > > > > > > > > > > > > > > > more
> > >> > > > > > > > > > > > > > > > >> complex workflow it would be
> certainly
> > >> > large.
> > >> > > > > > > > > > > > > > > > >> I think that this is more related to
> > how
> > >> we
> > >> > > are
> > >> > > > > > > handling
> > >> > > > > > > > > > JPA in
> > >> > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > code,
> > >> > > > > > > > > > > > > > > > >> in particular the mapping from model
> to
> > >> > entity
> > >> > > > > > > > (basically
> > >> > > > > > > > > > JPA is
> > >> > > > > > > > > > > > > > blind
> > >> > > > > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > >> has to update all nodes for every
> write
> > >> > > because
> > >> > > > it
> > >> > > > > > > > > believes
> > >> > > > > > > > > > the
> > >> > > > > > > > > > > > > > node has
> > >> > > > > > > > > > > > > > > > >> been updated, although it is not)
> than
> > an
> > >> > > issue
> > >> > > > in
> > >> > > > > > the
> > >> > > > > > > > > table
> > >> > > > > > > > > > > > > > definition.
> > >> > > > > > > > > > > > > > > > >> In fact, when using JPA, separating
> the
> > >> > server
> > >> > > > > model
> > >> > > > > > > > from
> > >> > > > > > > > > > the
> > >> > > > > > > > > > > > JPA
> > >> > > > > > > > > > > > > > > > entity is
> > >> > > > > > > > > > > > > > > > >> not a good idea, especially if the
> > entity
> > >> > > > contains
> > >> > > > > > > > > > collections.
> > >> > > > > > > > > > > > I
> > >> > > > > > > > > > > > > > will
> > >> > > > > > > > > > > > > > > > try
> > >> > > > > > > > > > > > > > > > >> to change that without breaking
> > anything.
> > >> > > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > > >> On Wed, Nov 22, 2023 at 12:10 PM
> > Enrique
> > >> > > > Gonzalez
> > >> > > > > > > > > Martinez <
> > >> > > > > > > > > > > > > > > > >> egonzalez@apache.org> wrote:
> > >> > > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > > >>> After the events split you now will
> > >> need to
> > >> > > > > create
> > >> > > > > > a
> > >> > > > > > > > node
> > >> > > > > > > > > > > > instance
> > >> > > > > > > > > > > > > > > > >>> model instance of making independent
> > >> from
> > >> > the
> > >> > > > > > process
> > >> > > > > > > > > > instance.
> > >> > > > > > > > > > > > > > > > >>> That should do the trick.
> > >> > > > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > > > >>> Regarding deleting/inserting it was
> > >> fixed
> > >> > at
> > >> > > > some
> > >> > > > > > > > point.
> > >> > > > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > > > >>> El mar, 21 nov 2023 a las 20:22,
> > >> Francisco
> > >> > > > Javier
> > >> > > > > > > > Tirado
> > >> > > > > > > > > > Sarti
> > >> > > > > > > > > > > > > > > > >>> (<ft...@redhat.com>) escribió:
> > >> > > > > > > > > > > > > > > > >>> >
> > >> > > > > > > > > > > > > > > > >>> > Hi Martin,
> > >> > > > > > > > > > > > > > > > >>> > I have a task to review
> performance
> > of
> > >> > > > > > > > > > > > > > > > >>> >
> > >> > > > > > > > > > > > > > > > >>> > ProcessInstanceNodeDataEventMerger
> > >> > > > > > > > > > > > > > > > >>> > My idea is to reduce the number of
> > >> delete
> > >> > > > > inserts
> > >> > > > > > > > when
> > >> > > > > > > > > > > > processing
> > >> > > > > > > > > > > > > > > > >>> events
> > >> > > > > > > > > > > > > > > > >>> > and try to do it incremental.
> > >> > > > > > > > > > > > > > > > >>> > That should improve performance.
> > >> > > > > > > > > > > > > > > > >>> > PS:
> > >> > > > > > > > > > > > > > > > >>> > I was planning to send an e-mail
> > >> tomorrow
> > >> > > > > > > announcing
> > >> > > > > > > > > > that in
> > >> > > > > > > > > > > > > > case you
> > >> > > > > > > > > > > > > > > > >>> were
> > >> > > > > > > > > > > > > > > > >>> > already working on a fix for
> that. I
> > >> > assume
> > >> > > > you
> > >> > > > > > are
> > >> > > > > > > > not
> > >> > > > > > > > > > and I
> > >> > > > > > > > > > > > > > would
> > >> > > > > > > > > > > > > > > > be
> > >> > > > > > > > > > > > > > > > >>> > sending a PR soon.
> > >> > > > > > > > > > > > > > > > >>> >
> > >> > > > > > > > > > > > > > > > >>> > On Tue, Nov 21, 2023 at 6:09 PM
> > Martin
> > >> > > Weiler
> > >> > > > > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > >> > > > > > > > > > > > > > > > >>> >
> > >> > > > > > > > > > > > > > > > >>> > wrote:
> > >> > > > > > > > > > > > > > > > >>> >
> > >> > > > > > > > > > > > > > > > >>> > > I looked into the new examples
> > using
> > >> > > > > data-index
> > >> > > > > > > > > > persistence
> > >> > > > > > > > > > > > > > addon -
> > >> > > > > > > > > > > > > > > > >>> Neus'
> > >> > > > > > > > > > > > > > > > >>> > > PR#1813 [1] for serverless and
> > >> Pere's
> > >> > > > branch
> > >> > > > > > [2]
> > >> > > > > > > > for
> > >> > > > > > > > > > > > workflow
> > >> > > > > > > > > > > > > > > > (great
> > >> > > > > > > > > > > > > > > > >>> job
> > >> > > > > > > > > > > > > > > > >>> > > both!) - and they work without
> > >> issues
> > >> > > using
> > >> > > > > > > single
> > >> > > > > > > > > > > > requests.
> > >> > > > > > > > > > > > > > > > >>> However, under
> > >> > > > > > > > > > > > > > > > >>> > > some load (I used 'ab' for
> testing
> > >> > with a
> > >> > > > > light
> > >> > > > > > > > > > > > concurrency of
> > >> > > > > > > > > > > > > > 10
> > >> > > > > > > > > > > > > > > > >>> parallel
> > >> > > > > > > > > > > > > > > > >>> > > requests) I ran into the
> following
> > >> > > > problems:
> > >> > > > > > > > > > > > > > > > >>> > >
> > >> > > > > > > > > > > > > > > > >>> > > (1) Large number of
> insert/delete
> > >> calls
> > >> > > > (eg.
> > >> > > > > > for
> > >> > > > > > > > > tables
> > >> > > > > > > > > > > > such as
> > >> > > > > > > > > > > > > > > > >>> nodes,
> > >> > > > > > > > > > > > > > > > >>> > > definitions, etc)
> > >> > > > > > > > > > > > > > > > >>> > >
> > >> > > > > > > > > > > > > > > > >>> > > (2) Hibernate
> > >> OptimisticLockExceptions
> > >> > /
> > >> > > > > > > > > > > > StaleStateExceptions
> > >> > > > > > > > > > > > > > > > >>> > >
> > >> > > > > > > > > > > > > > > > >>> > > (3) DB deadlocks
> > >> > > > > > > > > > > > > > > > >>> > >
> > >> > > > > > > > > > > > > > > > >>> > > (4) Error responses, slow
> response
> > >> > times
> > >> > > > > > > > > > > > > > > > >>> > >
> > >> > > > > > > > > > > > > > > > >>> > > The reason I am reaching out
> with
> > >> this
> > >> > > > topic
> > >> > > > > > here
> > >> > > > > > > > is
> > >> > > > > > > > > to
> > >> > > > > > > > > > > > find
> > >> > > > > > > > > > > > > > out if
> > >> > > > > > > > > > > > > > > > >>> we are
> > >> > > > > > > > > > > > > > > > >>> > > aware of this issue, and if
> > someone
> > >> is
> > >> > > > > already
> > >> > > > > > > > > looking
> > >> > > > > > > > > > > > into or
> > >> > > > > > > > > > > > > > > > being
> > >> > > > > > > > > > > > > > > > >>> > > assigned to it?
> > >> > > > > > > > > > > > > > > > >>> > >
> > >> > > > > > > > > > > > > > > > >>> > > Thanks,
> > >> > > > > > > > > > > > > > > > >>> > > Martin
> > >> > > > > > > > > > > > > > > > >>> > >
> > >> > > > > > > > > > > > > > > > >>> > > [1]
> > >> > > > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > >
> > >> > > > > > >
> > >> > https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > >> > > > > > > > > > > > > > > > >>> > > [2]
> > >> > > > > > > > > > > > > > > > >>> > >
> > >> > > > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > >> > > > > > > > > > > > > > > > >>> > >
> > >> > > > > > > > > > > > > > > > >>> > >
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > >
> > >> > > >
> > >> ---------------------------------------------------------------------
> > >> > > > > > > > > > > > > > > > >>> > > To unsubscribe, e-mail:
> > >> > > > > > > > > dev-unsubscribe@kie.apache.org
> > >> > > > > > > > > > > > > > > > >>> > > For additional commands, e-mail:
> > >> > > > > > > > > > dev-help@kie.apache.org
> > >> > > > > > > > > > > > > > > > >>> > >
> > >> > > > > > > > > > > > > > > > >>> > >
> > >> > > > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > >
> > >> >
> ---------------------------------------------------------------------
> > >> > > > > > > > > > > > > > > > >>> To unsubscribe, e-mail:
> > >> > > > > > > dev-unsubscribe@kie.apache.org
> > >> > > > > > > > > > > > > > > > >>> For additional commands, e-mail:
> > >> > > > > > > > dev-help@kie.apache.org
> > >> > > > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > >
> > >> > > >
> > >> ---------------------------------------------------------------------
> > >> > > > > > > > > > > > > > > > > To unsubscribe, e-mail:
> > >> > > > > > dev-unsubscribe@kie.apache.org
> > >> > > > > > > > > > > > > > > > > For additional commands, e-mail:
> > >> > > > > > > dev-help@kie.apache.org
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > >
> > >> >
> ---------------------------------------------------------------------
> > >> > > > > > > > > > > > > > To unsubscribe, e-mail:
> > >> > > dev-unsubscribe@kie.apache.org
> > >> > > > > > > > > > > > > > For additional commands, e-mail:
> > >> > > > dev-help@kie.apache.org
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > >
> > >> > > >
> > >> ---------------------------------------------------------------------
> > >> > > > > > > > > > > > To unsubscribe, e-mail:
> > >> dev-unsubscribe@kie.apache.org
> > >> > > > > > > > > > > > For additional commands, e-mail:
> > >> > dev-help@kie.apache.org
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > >
> > >> >
> ---------------------------------------------------------------------
> > >> > > > > > > > > > To unsubscribe, e-mail:
> > dev-unsubscribe@kie.apache.org
> > >> > > > > > > > > > For additional commands, e-mail:
> > >> dev-help@kie.apache.org
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > >
> > >> ---------------------------------------------------------------------
> > >> > > > > > > > > To unsubscribe, e-mail:
> dev-unsubscribe@kie.apache.org
> > >> > > > > > > > > For additional commands, e-mail:
> > dev-help@kie.apache.org
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > >> For additional commands, e-mail: dev-help@kie.apache.org
> > >>
> > >>
> >
>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Alex Porcelli <po...@apache.org>]

if data index was supposed to provide snapshot view of the process
instance… why do we keep it after the process instance is finished?


On Mon, Feb 19, 2024 at 7:12 AM Francisco Javier Tirado Sarti <
ftirados@redhat.com> wrote:

> Hi Martin.
> After taking a deeper look at this, I realize that the behaviour is the
> expected one.
> Runtimes DB does not track the completed process instance (that's what the
> JDBCProcessInstances warn is telling us), but DataIndex, as expected, is
> tracking it in processes and nodes table. And yes it will grow over time.
> What we need is some configurable purge mechanism for DataIndex, so it
> eventually removes older completed process instances.
>
> On Tue, Feb 13, 2024 at 12:59 PM Francisco Javier Tirado Sarti <
> ftirados@redhat.com> wrote:
>
> > Hi Martin,
> > Good catch!. Looks like the skipping performed for process instances is
> > not applied to node instances. Something we definitely need to review on
> > the runtimes side.
> >
> > On Mon, Feb 12, 2024 at 11:59 PM Martin Weiler <mw...@ibm.com.invalid>
> > wrote:
> >
> >> On a somewhat related note, testing a simple workflow (start -> script
> >> node -> end), I see the following messages in the logs:
> >> 2024-02-12 22:49:50,493 28758dde544c WARN
> >> [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> >> (executor-thread-3) Skipping create of process instance id:
> >> 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2
> >>
> >> So far, so good. And I'd expect to see no trace of this process in the
> >> database if I don't have data audit enabled.
> >>
> >> However, the 'processes' table contains a row with state=2, with related
> >> entries in the 'nodes' table. In a load test, I see these tables grow
> >> significantly over time. Am I missing something to have these entries
> >> cleaned up automatically?
> >>
> >> ________________________________________
> >> From: Martin Weiler <mw...@ibm.com.INVALID>
> >> Sent: Monday, February 12, 2024 3:40 PM
> >> To: dev@kie.apache.org
> >> Subject: [EXTERNAL] RE: [DISCUSSION] Performance issues with data-index
> >> persistence addon
> >>
> >> Thanks everyone for your input. Based on this discussion, I opened the
> >> following PR:
> >> https://github.com/apache/incubator-kie-kogito-apps/pull/1985
> >>
> >> With this change, the performance seems to be stable over time:
> >>
> >>
> https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
> >>
> >> Martin
> >>
> >> ________________________________________
> >> From: Gonzalo Muñoz <gm...@apache.org>
> >> Sent: Friday, February 9, 2024 9:42 AM
> >> To: dev@kie.apache.org
> >> Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with data-index
> >> persistence addon
> >>
> >> Great work Francisco,
> >> Martin, take a look at this link with some related tips (in case you
> find
> >> it useful):
> >> https://www.cybertec-postgresql.com/en/index-your-foreign-key/
> >>
> >> El vie, 9 feb 2024 a las 17:20, Francisco Javier Tirado Sarti (<
> >> ftirados@redhat.com>) escribió:
> >>
> >> > For the moment being, we will keep JPA till we exhaust all
> >> possibilities,
> >> > let's call switching from jpa to jdbc our hidden plan B ;)
> >> > I already told Martin, but in order everyone to know, just after
> writing
> >> > the previous email, I thought "what if Postgres is not automatically
> >> > indexing foreign keys like mysql?" and, eureka
> >> > Postgres doc
> >> > https://www.postgresql.org/docs/current/ddl-constraints.html
> >> > Mysql doc
> >> > https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> >> > These are the relevant excerpt
> >> >
> >> > *Postgresql*
> >> > *A foreign key must reference columns that either are a primary key or
> >> form
> >> > a unique constraint, or are columns from a non-partial unique index.
> >> This
> >> > means that the referenced columns always have an index to allow
> >> efficient
> >> > lookups on whether a referencing row has a match. Since a DELETE of a
> >> row
> >> > from the referenced table or an UPDATE of a referenced column will
> >> require
> >> > a scan of the referencing table for rows matching the old value, it is
> >> > often a good idea to index the referencing columns too. Because this
> is
> >> not
> >> > always needed, and there are many choices available on how to index,
> the
> >> > declaration of a foreign key constraint does not automatically create
> an
> >> > index on the referencing columns.*
> >> > *Mysql*
> >> > *MySQL requires that foreign key columns be indexed; if you create a
> >> table
> >> > with a foreign key constraint but no index on a given column, an index
> >> is
> >> > created. *
> >> >
> >> > So I asked Martin to especially create an index for
> process_instance_id
> >> > column on nodes table
> >> > I think that will fix the problem detected on the thread dump.
> >> > The simpler process test to verify queries are fine still stands,
> >> though ;)
> >> >
> >> >
> >> > On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <tz...@apache.org>
> >> wrote:
> >> >
> >> > > I always preferred pure JDBC over Hibernate myself, just for the
> sake
> >> of
> >> > > control of what is happening :) So I would not -1 that myself.
> >> > >
> >> > > Tibor
> >> > >
> >> > > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado Sarti <
> >> > > ftirados@redhat.com>
> >> > > napísal(a):
> >> > >
> >> > > > Hi,
> >> > > > Usually I do not want to talk about work in progress because
> >> > preliminary
> >> > > > conclusions are pretty volatile but, well, there are a couple of
> >> things
> >> > > > that can be concluded from the really valuable information that
> >> Martin
> >> > > > provided.
> >> > > > 1) In order to be able to determine if the number of statements is
> >> > larger
> >> > > > than expected, I asked Martin to test with a simpler process
> >> > definition.
> >> > > > One with just three nodes: start, script and end. The script one
> >> should
> >> > > > change just one variable. This way we can analyze if the number of
> >> > > queries
> >> > > > is the expected one. From the single log (audit was activated
> them)
> >> my
> >> > > > conclusion is that the number of insert/updates over processes and
> >> > nodes
> >> > > > (there a lot over task, that I will prefer to skip for now, baby
> >> steps)
> >> > > is
> >> > > > the expected one.
> >> > > > 2) Analysing the thread dump, we see around 15 threads executing
> >> this
> >> > > line
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> >> > > > so its pretty clear the code to be optimized ;). I'm evaluating
> >> > > > possibilities within JPA/Hibernate, but I'm starting to think that
> >> it
> >> > > might
> >> > > > be better to switch to JDBC and skip hibernate. Our lives will be
> >> > > simpler,
> >> > > > especially with a schema relatively simple like ours (that will be
> >> my
> >> > > > recommendation if I was an external consultant)
> >> > > >
> >> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <tzimanyi@apache.org
> >
> >> > > wrote:
> >> > > >
> >> > > > > Hi,
> >> > > > >
> >> > > > > this will be a bit off-topic. However as far as performance, I
> >> think
> >> > we
> >> > > > > should think about that we have string primary keys (IDs). I
> would
> >> > > expect
> >> > > > > the database systems are much better with indexing numeric keys
> >> than
> >> > > > > strings. I remember from the past, when I was working with DBs,
> >> that
> >> > > > using
> >> > > > > strings as keys or indexes was a discouraged practice.
> >> > > > >
> >> > > > > Best regards,
> >> > > > > Tibor
> >> > > > >
> >> > > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler <mweiler@ibm.com.invalid
> >
> >> > > > > napísal(a):
> >> > > > >
> >> > > > > > I changed the test to use MongoDB [1] and I don't see a
> >> performance
> >> > > > > > degradation with this setup [2].
> >> > > > > >
> >> > > > > > Please keep us posted of your findings. Thanks!
> >> > > > > >
> >> > > > > > Martin
> >> > > > > >
> >> > > > > > [1]
> >> > > > >
> >> >
> https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> >> > > > > > [2]
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> >> > > > > >
> >> > > > > > ________________________________________
> >> > > > > > From: Francisco Javier Tirado Sarti <ft...@redhat.com>
> >> > > > > > Sent: Wednesday, February 7, 2024 11:40 AM
> >> > > > > > To: dev@kie.apache.org
> >> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> >> > > data-index
> >> > > > > > persistence addon
> >> > > > > >
> >> > > > > > yes, it can be index degradation because of size, but I
> believe
> >> (I
> >> > > > might
> >> > > > > be
> >> > > > > > wrong) the db is too small (yet) for that.
> >> > > > > > But, eventually, Postgres, when the DB is huge enough,
> >> unavoidably
> >> > > will
> >> > > > > > behave like the graphic that Martin sent.
> >> > > > > > Since I believe we are not huge enough (yet), lets rule out
> >> another
> >> > > > issue
> >> > > > > > by analysing the sql logs (I requested those to Martin offline
> >> and
> >> > he
> >> > > > is
> >> > > > > > going to kindly collect them).
> >> > > > > > Also Im curious to know if Mongo behave in the same way.
> >> > > > > >
> >> > > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez Martinez <
> >> > > > > > egonzalez@apache.org> wrote:
> >> > > > > >
> >> > > > > > > Hi Francisco,
> >> > > > > > > I would highly recommend to check indexes and how the
> updates
> >> > work
> >> > > in
> >> > > > > > data
> >> > > > > > > index to avoid full scan table and lock the full table. Some
> >> db
> >> > are
> >> > > > > very
> >> > > > > > > sensitive to that.
> >> > > > > > >
> >> > > > > > > El mié, 7 feb 2024, 18:41, Francisco Javier Tirado Sarti <
> >> > > > > > > ftirados@redhat.com> escribió:
> >> > > > > > >
> >> > > > > > > > Hi Martin,
> >> > > > > > > > While I analyze the data, let me ask you if it is possible
> >> to
> >> > > > perform
> >> > > > > > > > another check (similar in a way to disabling data-index
> like
> >> > you
> >> > > > do)
> >> > > > > > Can
> >> > > > > > > > you switch to MongoDB persistence and check if the same
> >> > > degradation
> >> > > > > > that
> >> > > > > > > is
> >> > > > > > > > there for postgres remains?
> >> > > > > > > > I do not know if this is feasible but will certainly
> >> indicate
> >> > the
> >> > > > > > problem
> >> > > > > > > > is on the postgres storage layer and I do not have a clear
> >> > > > prediction
> >> > > > > > of
> >> > > > > > > > what we will see when doing this switch.
> >> > > > > > > >
> >> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler
> >> > > > <mweiler@ibm.com.invalid
> >> > > > > >
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Hi Francisco,
> >> > > > > > > > >
> >> > > > > > > > > thanks for your work on this important topic!
> >> > > > > > > > >
> >> > > > > > > > > I would like to share some test results here, which
> might
> >> > help
> >> > > to
> >> > > > > > > improve
> >> > > > > > > > > the codebase even further. I am using the jmeter based
> >> test
> >> > > case
> >> > > > > from
> >> > > > > > > > Pere
> >> > > > > > > > > and Enrique (thanks guys!) [1] which uses a load of 30
> >> > threads
> >> > > to
> >> > > > > > > > >
> >> > > > > > > > > 1) start a new process instance (POST)
> >> > > > > > > > > 2) retrieve tasks for a user (GET)
> >> > > > > > > > > 3) fetches task details (GET)
> >> > > > > > > > > 4) complete a task (POST)
> >> > > > > > > > > 5) execute a query on data-audit
> >> > > > > > > > >
> >> > > > > > > > > With this test setup, I noticed that the performance for
> >> the
> >> > > POST
> >> > > > > > > > > requests, in particular the one to start a new process
> >> > > instance,
> >> > > > > > > degrades
> >> > > > > > > > > over time - see graph [2]. If I run the same test
> without
> >> > > > > data-index,
> >> > > > > > > > then
> >> > > > > > > > > there is no such performance degradation [3]. You can
> >> find a
> >> > > > thread
> >> > > > > > > dump
> >> > > > > > > > > captured a few minutes into the first test here [4] that
> >> > might
> >> > > > help
> >> > > > > > to
> >> > > > > > > > see
> >> > > > > > > > > some of the contention points.
> >> > > > > > > > >
> >> > > > > > > > > I'd appreciate if you could take a look and see if there
> >> is
> >> > > > > something
> >> > > > > > > > that
> >> > > > > > > > > can be further improved based on your previous work. If
> >> you
> >> > > need
> >> > > > > any
> >> > > > > > > > > additional data, let me know, but otherwise it is
> >> > > straightforward
> >> > > > > to
> >> > > > > > > run
> >> > > > > > > > > the jmeter test as well.
> >> > > > > > > > >
> >> > > > > > > > > Thanks,
> >> > > > > > > > > Martin
> >> > > > > > > > >
> >> > > > > > > > > [1]
> >> https://github.com/pefernan/job-service-refactor-test/
> >> > > > > > > > > [2]
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> >> > > > > > > > > [3]
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> >> > > > > > > > > [4]
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> >> > > > > > > > >
> >> > > > > > > > > ________________________________________
> >> > > > > > > > > From: Francisco Javier Tirado Sarti <
> ftirados@redhat.com>
> >> > > > > > > > > Sent: Wednesday, January 17, 2024 9:13 AM
> >> > > > > > > > > To: dev@kie.apache.org
> >> > > > > > > > > Cc: Pere Fernandez Perez
> >> > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues
> >> with
> >> > > > > > data-index
> >> > > > > > > > > persistence addon
> >> > > > > > > > >
> >> > > > > > > > > Hi Alex,
> >> > > > > > > > > I did not take times (which depends on a number of
> >> variables
> >> > > that
> >> > > > > > > > > drastically change between environments), but verify
> that
> >> the
> >> > > > > number
> >> > > > > > of
> >> > > > > > > > > updates has been reduced drastically without losing
> >> > > > functionality,
> >> > > > > > > which
> >> > > > > > > > is
> >> > > > > > > > > objectively a good thing. If before the change, for
> every
> >> > node
> >> > > > > > > executed,
> >> > > > > > > > we
> >> > > > > > > > > have an update for every node previously executed, so
> if a
> >> > > > process
> >> > > > > > have
> >> > > > > > > > 50
> >> > > > > > > > > nodes to execute, we were performing nearly 50*51/2
> >> updates,
> >> > > > which
> >> > > > > > > gives
> >> > > > > > > > us
> >> > > > > > > > > a total of  1275 updates, now we have just one for every
> >> node
> >> > > > being
> >> > > > > > > > > executed, implying a total of 50 updates.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli <
> >> > > alex@porcelli.me>
> >> > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Francisco,
> >> > > > > > > > > >
> >> > > > > > > > > > I noticed that your PR has been merged, but I was
> >> expecting
> >> > > (at
> >> > > > > > least
> >> > > > > > > > > > was my understanding from this thread) that before
> >> merging
> >> > > some
> >> > > > > > > > > > benchmark data would be shared in advance - to assess
> >> the
> >> > > > > > > cost/benefit
> >> > > > > > > > > > of such a decent size change.
> >> > > > > > > > > >
> >> > > > > > > > > > Do you have any information to share?
> >> > > > > > > > > >
> >> > > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM Francisco Javier
> Tirado
> >> > Sarti
> >> > > > > > > > > > <ft...@redhat.com> wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > Yes, as intended, now we have one select and one
> >> > > > insert/update
> >> > > > > > per
> >> > > > > > > > node
> >> > > > > > > > > > > event.
> >> > > > > > > > > > > I moved the PR as ready for review and give @Pere
> >> > Fernandez
> >> > > > > Perez
> >> > > > > > > > > > > <pe...@redhat.com> permission to the branch so
> he
> >> can
> >> > > > edit
> >> > > > > it
> >> > > > > > > in
> >> > > > > > > > > the
> >> > > > > > > > > > > next two weeks (Ill be on PTO)  if desired, before
> >> > merging.
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex Porcelli <
> >> > > > > alex@porcelli.me>
> >> > > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > > Cool, thank you Francisco!
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Did you manage to get some preliminary data about
> >> > > > > improvements?
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM Francisco Javier
> >> > Tirado
> >> > > > > Sarti
> >> > > > > > > > > > > > <ft...@redhat.com> wrote:
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Yes, after some delay because of quarkus 3
> >> migration.
> >> > > Im
> >> > > > > > > refining
> >> > > > > > > > > > this
> >> > > > > > > > > > > > > draft PR
> >> > > > > > > > > > > > >
> >> > > > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex Porcelli <
> >> > > > > > > alex@porcelli.me>
> >> > > > > > > > > > wrote:
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Any update or new findings on this topic?
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM Francisco
> Javier
> >> > > Tirado
> >> > > > > > Sarti
> >> > > > > > > > > > > > > > <ft...@redhat.com> wrote:
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Hi Alex,
> >> > > > > > > > > > > > > > > After considering different options to
> improve
> >> > > > > > performance,
> >> > > > > > > > we
> >> > > > > > > > > > feel
> >> > > > > > > > > > > > that
> >> > > > > > > > > > > > > > it
> >> > > > > > > > > > > > > > > is time to "partially" move away from the
> >> current
> >> > > Map
> >> > > > > > style
> >> > > > > > > > > > > > interface (
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> >> > > > > > > > > > > > > > )
> >> > > > > > > > > > > > > > > which was shared with Trusty, to one more
> >> > suitable
> >> > > > for
> >> > > > > > > usage
> >> > > > > > > > > > with a
> >> > > > > > > > > > > > > > > relational DB like postgresql (but still
> >> > compatible
> >> > > > > with
> >> > > > > > > big
> >> > > > > > > > > > table
> >> > > > > > > > > > > > dbs).
> >> > > > > > > > > > > > > > > The idea will be to replace generic Storage
> >> > > interface
> >> > > > > by
> >> > > > > > > four
> >> > > > > > > > > > > > specific
> >> > > > > > > > > > > > > > > interfaces (which will inherit from a common
> >> one
> >> > > that
> >> > > > > > keeps
> >> > > > > > > > the
> >> > > > > > > > > > query
> >> > > > > > > > > > > > > > part
> >> > > > > > > > > > > > > > > at is it. with get and query methods), that
> >> will
> >> > > > > include
> >> > > > > > > the
> >> > > > > > > > > > required
> >> > > > > > > > > > > > > > > modification operations for the four
> DataIndex
> >> > > > > "domains":
> >> > > > > > > > > > > > > > processinstance,
> >> > > > > > > > > > > > > > > usertask, processdefinitions and jobs. Those
> >> > > > interfaces
> >> > > > > > > will
> >> > > > > > > > > > define
> >> > > > > > > > > > > > > > methods
> >> > > > > > > > > > > > > > > like addNode, addVariable, updateTask,
> >> > > > > addAttachment.....
> >> > > > > > > > that
> >> > > > > > > > > > will
> >> > > > > > > > > > > > allow
> >> > > > > > > > > > > > > > > the persistent layer implementation  to just
> >> > update
> >> > > > the
> >> > > > > > > > needed
> >> > > > > > > > > > info
> >> > > > > > > > > > > > in
> >> > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > DB  (for example, for addNode in Postgres,
> >> just
> >> > > > insert
> >> > > > > a
> >> > > > > > > row
> >> > > > > > > > > into
> >> > > > > > > > > > > > nodes
> >> > > > > > > > > > > > > > > table, for addNode in Mongo, basically the
> >> same
> >> > > > atomic
> >> > > > > > > upsert
> >> > > > > > > > > > > > operation
> >> > > > > > > > > > > > > > > that is currently done). Therefore, we
> >> increase
> >> > > > > > performance
> >> > > > > > > > for
> >> > > > > > > > > > > > Postgres
> >> > > > > > > > > > > > > > > and keep the current one for Mongo. The
> >> current
> >> > DB
> >> > > > > > schemas
> >> > > > > > > > > won't
> >> > > > > > > > > > be
> >> > > > > > > > > > > > > > > touched.
> >> > > > > > > > > > > > > > > Since the code change is large, I do not
> think
> >> > I'll
> >> > > > be
> >> > > > > > able
> >> > > > > > > > to
> >> > > > > > > > > > have
> >> > > > > > > > > > > > the
> >> > > > > > > > > > > > > > PR
> >> > > > > > > > > > > > > > > ready till next week.
> >> > > > > > > > > > > > > > > But before starting, please let me know if
> >> that
> >> > > > > approach
> >> > > > > > is
> >> > > > > > > > > fine
> >> > > > > > > > > > for
> >> > > > > > > > > > > > you.
> >> > > > > > > > > > > > > > > Best regards.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 6:55 PM Alex
> Porcelli
> >> <
> >> > > > > > > > > alex@porcelli.me>
> >> > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > Thank you Francisco to getting deeper on
> >> this…
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > Looking forward to see the results of your
> >> > > > suggested
> >> > > > > > > > > > improvements.
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 9:40 AM Francisco
> >> > Javier
> >> > > > > Tirado
> >> > > > > > > > > Sarti <
> >> > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > I forgot to attach the queries
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 3:04 PM
> Francisco
> >> > > Javier
> >> > > > > > Tirado
> >> > > > > > > > > > Sarti <
> >> > > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > >> Hi,
> >> > > > > > > > > > > > > > > > >> A brief update on this topic.
> >> > > > > > > > > > > > > > > > >> After doing a simple test with example
> >> > > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> >> > > > > > > > > > > > > > > > ,
> >> > > > > > > > > > > > > > > > >> the number of updates over Nodes table
> is
> >> > n*n,
> >> > > > so
> >> > > > > we
> >> > > > > > > > > manage
> >> > > > > > > > > > to
> >> > > > > > > > > > > > > > obtain a
> >> > > > > > > > > > > > > > > > >> perfect quadratic performance
> >> degradation.
> >> > The
> >> > > > > > problem
> >> > > > > > > > is
> >> > > > > > > > > > worse
> >> > > > > > > > > > > > in
> >> > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > case
> >> > > > > > > > > > > > > > > > >> of Serverless Workflow than in BPMN
> >> because
> >> > we
> >> > > > the
> >> > > > > > > > number
> >> > > > > > > > > of
> >> > > > > > > > > > > > nodes
> >> > > > > > > > > > > > > > is
> >> > > > > > > > > > > > > > > > >> greater than the number of states. In
> >> that
> >> > > > > example N
> >> > > > > > > is
> >> > > > > > > > > 16,
> >> > > > > > > > > > but
> >> > > > > > > > > > > > for
> >> > > > > > > > > > > > > > a
> >> > > > > > > > > > > > > > > > more
> >> > > > > > > > > > > > > > > > >> complex workflow it would be certainly
> >> > large.
> >> > > > > > > > > > > > > > > > >> I think that this is more related to
> how
> >> we
> >> > > are
> >> > > > > > > handling
> >> > > > > > > > > > JPA in
> >> > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > code,
> >> > > > > > > > > > > > > > > > >> in particular the mapping from model to
> >> > entity
> >> > > > > > > > (basically
> >> > > > > > > > > > JPA is
> >> > > > > > > > > > > > > > blind
> >> > > > > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > >> has to update all nodes for every write
> >> > > because
> >> > > > it
> >> > > > > > > > > believes
> >> > > > > > > > > > the
> >> > > > > > > > > > > > > > node has
> >> > > > > > > > > > > > > > > > >> been updated, although it is not) than
> an
> >> > > issue
> >> > > > in
> >> > > > > > the
> >> > > > > > > > > table
> >> > > > > > > > > > > > > > definition.
> >> > > > > > > > > > > > > > > > >> In fact, when using JPA, separating the
> >> > server
> >> > > > > model
> >> > > > > > > > from
> >> > > > > > > > > > the
> >> > > > > > > > > > > > JPA
> >> > > > > > > > > > > > > > > > entity is
> >> > > > > > > > > > > > > > > > >> not a good idea, especially if the
> entity
> >> > > > contains
> >> > > > > > > > > > collections.
> >> > > > > > > > > > > > I
> >> > > > > > > > > > > > > > will
> >> > > > > > > > > > > > > > > > try
> >> > > > > > > > > > > > > > > > >> to change that without breaking
> anything.
> >> > > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > > >> On Wed, Nov 22, 2023 at 12:10 PM
> Enrique
> >> > > > Gonzalez
> >> > > > > > > > > Martinez <
> >> > > > > > > > > > > > > > > > >> egonzalez@apache.org> wrote:
> >> > > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > > >>> After the events split you now will
> >> need to
> >> > > > > create
> >> > > > > > a
> >> > > > > > > > node
> >> > > > > > > > > > > > instance
> >> > > > > > > > > > > > > > > > >>> model instance of making independent
> >> from
> >> > the
> >> > > > > > process
> >> > > > > > > > > > instance.
> >> > > > > > > > > > > > > > > > >>> That should do the trick.
> >> > > > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > > > >>> Regarding deleting/inserting it was
> >> fixed
> >> > at
> >> > > > some
> >> > > > > > > > point.
> >> > > > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > > > >>> El mar, 21 nov 2023 a las 20:22,
> >> Francisco
> >> > > > Javier
> >> > > > > > > > Tirado
> >> > > > > > > > > > Sarti
> >> > > > > > > > > > > > > > > > >>> (<ft...@redhat.com>) escribió:
> >> > > > > > > > > > > > > > > > >>> >
> >> > > > > > > > > > > > > > > > >>> > Hi Martin,
> >> > > > > > > > > > > > > > > > >>> > I have a task to review performance
> of
> >> > > > > > > > > > > > > > > > >>> >
> >> > > > > > > > > > > > > > > > >>> > ProcessInstanceNodeDataEventMerger
> >> > > > > > > > > > > > > > > > >>> > My idea is to reduce the number of
> >> delete
> >> > > > > inserts
> >> > > > > > > > when
> >> > > > > > > > > > > > processing
> >> > > > > > > > > > > > > > > > >>> events
> >> > > > > > > > > > > > > > > > >>> > and try to do it incremental.
> >> > > > > > > > > > > > > > > > >>> > That should improve performance.
> >> > > > > > > > > > > > > > > > >>> > PS:
> >> > > > > > > > > > > > > > > > >>> > I was planning to send an e-mail
> >> tomorrow
> >> > > > > > > announcing
> >> > > > > > > > > > that in
> >> > > > > > > > > > > > > > case you
> >> > > > > > > > > > > > > > > > >>> were
> >> > > > > > > > > > > > > > > > >>> > already working on a fix for that. I
> >> > assume
> >> > > > you
> >> > > > > > are
> >> > > > > > > > not
> >> > > > > > > > > > and I
> >> > > > > > > > > > > > > > would
> >> > > > > > > > > > > > > > > > be
> >> > > > > > > > > > > > > > > > >>> > sending a PR soon.
> >> > > > > > > > > > > > > > > > >>> >
> >> > > > > > > > > > > > > > > > >>> > On Tue, Nov 21, 2023 at 6:09 PM
> Martin
> >> > > Weiler
> >> > > > > > > > > > > > > > > > <mweiler@ibm.com.invalid
> >> > > > > > > > > > > > > > > > >>> >
> >> > > > > > > > > > > > > > > > >>> > wrote:
> >> > > > > > > > > > > > > > > > >>> >
> >> > > > > > > > > > > > > > > > >>> > > I looked into the new examples
> using
> >> > > > > data-index
> >> > > > > > > > > > persistence
> >> > > > > > > > > > > > > > addon -
> >> > > > > > > > > > > > > > > > >>> Neus'
> >> > > > > > > > > > > > > > > > >>> > > PR#1813 [1] for serverless and
> >> Pere's
> >> > > > branch
> >> > > > > > [2]
> >> > > > > > > > for
> >> > > > > > > > > > > > workflow
> >> > > > > > > > > > > > > > > > (great
> >> > > > > > > > > > > > > > > > >>> job
> >> > > > > > > > > > > > > > > > >>> > > both!) - and they work without
> >> issues
> >> > > using
> >> > > > > > > single
> >> > > > > > > > > > > > requests.
> >> > > > > > > > > > > > > > > > >>> However, under
> >> > > > > > > > > > > > > > > > >>> > > some load (I used 'ab' for testing
> >> > with a
> >> > > > > light
> >> > > > > > > > > > > > concurrency of
> >> > > > > > > > > > > > > > 10
> >> > > > > > > > > > > > > > > > >>> parallel
> >> > > > > > > > > > > > > > > > >>> > > requests) I ran into the following
> >> > > > problems:
> >> > > > > > > > > > > > > > > > >>> > >
> >> > > > > > > > > > > > > > > > >>> > > (1) Large number of insert/delete
> >> calls
> >> > > > (eg.
> >> > > > > > for
> >> > > > > > > > > tables
> >> > > > > > > > > > > > such as
> >> > > > > > > > > > > > > > > > >>> nodes,
> >> > > > > > > > > > > > > > > > >>> > > definitions, etc)
> >> > > > > > > > > > > > > > > > >>> > >
> >> > > > > > > > > > > > > > > > >>> > > (2) Hibernate
> >> OptimisticLockExceptions
> >> > /
> >> > > > > > > > > > > > StaleStateExceptions
> >> > > > > > > > > > > > > > > > >>> > >
> >> > > > > > > > > > > > > > > > >>> > > (3) DB deadlocks
> >> > > > > > > > > > > > > > > > >>> > >
> >> > > > > > > > > > > > > > > > >>> > > (4) Error responses, slow response
> >> > times
> >> > > > > > > > > > > > > > > > >>> > >
> >> > > > > > > > > > > > > > > > >>> > > The reason I am reaching out with
> >> this
> >> > > > topic
> >> > > > > > here
> >> > > > > > > > is
> >> > > > > > > > > to
> >> > > > > > > > > > > > find
> >> > > > > > > > > > > > > > out if
> >> > > > > > > > > > > > > > > > >>> we are
> >> > > > > > > > > > > > > > > > >>> > > aware of this issue, and if
> someone
> >> is
> >> > > > > already
> >> > > > > > > > > looking
> >> > > > > > > > > > > > into or
> >> > > > > > > > > > > > > > > > being
> >> > > > > > > > > > > > > > > > >>> > > assigned to it?
> >> > > > > > > > > > > > > > > > >>> > >
> >> > > > > > > > > > > > > > > > >>> > > Thanks,
> >> > > > > > > > > > > > > > > > >>> > > Martin
> >> > > > > > > > > > > > > > > > >>> > >
> >> > > > > > > > > > > > > > > > >>> > > [1]
> >> > > > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > >
> >> > > > > > >
> >> > https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> >> > > > > > > > > > > > > > > > >>> > > [2]
> >> > > > > > > > > > > > > > > > >>> > >
> >> > > > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> >> > > > > > > > > > > > > > > > >>> > >
> >> > > > > > > > > > > > > > > > >>> > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > >
> >> > > >
> >> ---------------------------------------------------------------------
> >> > > > > > > > > > > > > > > > >>> > > To unsubscribe, e-mail:
> >> > > > > > > > > dev-unsubscribe@kie.apache.org
> >> > > > > > > > > > > > > > > > >>> > > For additional commands, e-mail:
> >> > > > > > > > > > dev-help@kie.apache.org
> >> > > > > > > > > > > > > > > > >>> > >
> >> > > > > > > > > > > > > > > > >>> > >
> >> > > > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > >
> >> > ---------------------------------------------------------------------
> >> > > > > > > > > > > > > > > > >>> To unsubscribe, e-mail:
> >> > > > > > > dev-unsubscribe@kie.apache.org
> >> > > > > > > > > > > > > > > > >>> For additional commands, e-mail:
> >> > > > > > > > dev-help@kie.apache.org
> >> > > > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > >
> >> > > >
> >> ---------------------------------------------------------------------
> >> > > > > > > > > > > > > > > > > To unsubscribe, e-mail:
> >> > > > > > dev-unsubscribe@kie.apache.org
> >> > > > > > > > > > > > > > > > > For additional commands, e-mail:
> >> > > > > > > dev-help@kie.apache.org
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > >
> >> > ---------------------------------------------------------------------
> >> > > > > > > > > > > > > > To unsubscribe, e-mail:
> >> > > dev-unsubscribe@kie.apache.org
> >> > > > > > > > > > > > > > For additional commands, e-mail:
> >> > > > dev-help@kie.apache.org
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > >
> >> > > >
> >> ---------------------------------------------------------------------
> >> > > > > > > > > > > > To unsubscribe, e-mail:
> >> dev-unsubscribe@kie.apache.org
> >> > > > > > > > > > > > For additional commands, e-mail:
> >> > dev-help@kie.apache.org
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > >
> >> > ---------------------------------------------------------------------
> >> > > > > > > > > > To unsubscribe, e-mail:
> dev-unsubscribe@kie.apache.org
> >> > > > > > > > > > For additional commands, e-mail:
> >> dev-help@kie.apache.org
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > >
> >> ---------------------------------------------------------------------
> >> > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> >> > > > > > > > > For additional commands, e-mail:
> dev-help@kie.apache.org
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> >> For additional commands, e-mail: dev-help@kie.apache.org
> >>
> >>
>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Francisco Javier Tirado Sarti <ft...@redhat.com>]

Hi Martin.
After taking a deeper look at this, I realize that the behaviour is the
expected one.
Runtimes DB does not track the completed process instance (that's what the
JDBCProcessInstances warn is telling us), but DataIndex, as expected, is
tracking it in processes and nodes table. And yes it will grow over time.
What we need is some configurable purge mechanism for DataIndex, so it
eventually removes older completed process instances.

On Tue, Feb 13, 2024 at 12:59 PM Francisco Javier Tirado Sarti <
ftirados@redhat.com> wrote:

> Hi Martin,
> Good catch!. Looks like the skipping performed for process instances is
> not applied to node instances. Something we definitely need to review on
> the runtimes side.
>
> On Mon, Feb 12, 2024 at 11:59 PM Martin Weiler <mw...@ibm.com.invalid>
> wrote:
>
>> On a somewhat related note, testing a simple workflow (start -> script
>> node -> end), I see the following messages in the logs:
>> 2024-02-12 22:49:50,493 28758dde544c WARN
>> [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
>> (executor-thread-3) Skipping create of process instance id:
>> 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2
>>
>> So far, so good. And I'd expect to see no trace of this process in the
>> database if I don't have data audit enabled.
>>
>> However, the 'processes' table contains a row with state=2, with related
>> entries in the 'nodes' table. In a load test, I see these tables grow
>> significantly over time. Am I missing something to have these entries
>> cleaned up automatically?
>>
>> ________________________________________
>> From: Martin Weiler <mw...@ibm.com.INVALID>
>> Sent: Monday, February 12, 2024 3:40 PM
>> To: dev@kie.apache.org
>> Subject: [EXTERNAL] RE: [DISCUSSION] Performance issues with data-index
>> persistence addon
>>
>> Thanks everyone for your input. Based on this discussion, I opened the
>> following PR:
>> https://github.com/apache/incubator-kie-kogito-apps/pull/1985
>>
>> With this change, the performance seems to be stable over time:
>>
>> https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
>>
>> Martin
>>
>> ________________________________________
>> From: Gonzalo Muñoz <gm...@apache.org>
>> Sent: Friday, February 9, 2024 9:42 AM
>> To: dev@kie.apache.org
>> Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with data-index
>> persistence addon
>>
>> Great work Francisco,
>> Martin, take a look at this link with some related tips (in case you find
>> it useful):
>> https://www.cybertec-postgresql.com/en/index-your-foreign-key/
>>
>> El vie, 9 feb 2024 a las 17:20, Francisco Javier Tirado Sarti (<
>> ftirados@redhat.com>) escribió:
>>
>> > For the moment being, we will keep JPA till we exhaust all
>> possibilities,
>> > let's call switching from jpa to jdbc our hidden plan B ;)
>> > I already told Martin, but in order everyone to know, just after writing
>> > the previous email, I thought "what if Postgres is not automatically
>> > indexing foreign keys like mysql?" and, eureka
>> > Postgres doc
>> > https://www.postgresql.org/docs/current/ddl-constraints.html
>> > Mysql doc
>> > https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
>> > These are the relevant excerpt
>> >
>> > *Postgresql*
>> > *A foreign key must reference columns that either are a primary key or
>> form
>> > a unique constraint, or are columns from a non-partial unique index.
>> This
>> > means that the referenced columns always have an index to allow
>> efficient
>> > lookups on whether a referencing row has a match. Since a DELETE of a
>> row
>> > from the referenced table or an UPDATE of a referenced column will
>> require
>> > a scan of the referencing table for rows matching the old value, it is
>> > often a good idea to index the referencing columns too. Because this is
>> not
>> > always needed, and there are many choices available on how to index, the
>> > declaration of a foreign key constraint does not automatically create an
>> > index on the referencing columns.*
>> > *Mysql*
>> > *MySQL requires that foreign key columns be indexed; if you create a
>> table
>> > with a foreign key constraint but no index on a given column, an index
>> is
>> > created. *
>> >
>> > So I asked Martin to especially create an index for process_instance_id
>> > column on nodes table
>> > I think that will fix the problem detected on the thread dump.
>> > The simpler process test to verify queries are fine still stands,
>> though ;)
>> >
>> >
>> > On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <tz...@apache.org>
>> wrote:
>> >
>> > > I always preferred pure JDBC over Hibernate myself, just for the sake
>> of
>> > > control of what is happening :) So I would not -1 that myself.
>> > >
>> > > Tibor
>> > >
>> > > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado Sarti <
>> > > ftirados@redhat.com>
>> > > napísal(a):
>> > >
>> > > > Hi,
>> > > > Usually I do not want to talk about work in progress because
>> > preliminary
>> > > > conclusions are pretty volatile but, well, there are a couple of
>> things
>> > > > that can be concluded from the really valuable information that
>> Martin
>> > > > provided.
>> > > > 1) In order to be able to determine if the number of statements is
>> > larger
>> > > > than expected, I asked Martin to test with a simpler process
>> > definition.
>> > > > One with just three nodes: start, script and end. The script one
>> should
>> > > > change just one variable. This way we can analyze if the number of
>> > > queries
>> > > > is the expected one. From the single log (audit was activated them)
>> my
>> > > > conclusion is that the number of insert/updates over processes and
>> > nodes
>> > > > (there a lot over task, that I will prefer to skip for now, baby
>> steps)
>> > > is
>> > > > the expected one.
>> > > > 2) Analysing the thread dump, we see around 15 threads executing
>> this
>> > > line
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
>> > > > so its pretty clear the code to be optimized ;). I'm evaluating
>> > > > possibilities within JPA/Hibernate, but I'm starting to think that
>> it
>> > > might
>> > > > be better to switch to JDBC and skip hibernate. Our lives will be
>> > > simpler,
>> > > > especially with a schema relatively simple like ours (that will be
>> my
>> > > > recommendation if I was an external consultant)
>> > > >
>> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <tz...@apache.org>
>> > > wrote:
>> > > >
>> > > > > Hi,
>> > > > >
>> > > > > this will be a bit off-topic. However as far as performance, I
>> think
>> > we
>> > > > > should think about that we have string primary keys (IDs). I would
>> > > expect
>> > > > > the database systems are much better with indexing numeric keys
>> than
>> > > > > strings. I remember from the past, when I was working with DBs,
>> that
>> > > > using
>> > > > > strings as keys or indexes was a discouraged practice.
>> > > > >
>> > > > > Best regards,
>> > > > > Tibor
>> > > > >
>> > > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler <mw...@ibm.com.invalid>
>> > > > > napísal(a):
>> > > > >
>> > > > > > I changed the test to use MongoDB [1] and I don't see a
>> performance
>> > > > > > degradation with this setup [2].
>> > > > > >
>> > > > > > Please keep us posted of your findings. Thanks!
>> > > > > >
>> > > > > > Martin
>> > > > > >
>> > > > > > [1]
>> > > > >
>> > https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
>> > > > > > [2]
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
>> > > > > >
>> > > > > > ________________________________________
>> > > > > > From: Francisco Javier Tirado Sarti <ft...@redhat.com>
>> > > > > > Sent: Wednesday, February 7, 2024 11:40 AM
>> > > > > > To: dev@kie.apache.org
>> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
>> > > data-index
>> > > > > > persistence addon
>> > > > > >
>> > > > > > yes, it can be index degradation because of size, but I believe
>> (I
>> > > > might
>> > > > > be
>> > > > > > wrong) the db is too small (yet) for that.
>> > > > > > But, eventually, Postgres, when the DB is huge enough,
>> unavoidably
>> > > will
>> > > > > > behave like the graphic that Martin sent.
>> > > > > > Since I believe we are not huge enough (yet), lets rule out
>> another
>> > > > issue
>> > > > > > by analysing the sql logs (I requested those to Martin offline
>> and
>> > he
>> > > > is
>> > > > > > going to kindly collect them).
>> > > > > > Also Im curious to know if Mongo behave in the same way.
>> > > > > >
>> > > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez Martinez <
>> > > > > > egonzalez@apache.org> wrote:
>> > > > > >
>> > > > > > > Hi Francisco,
>> > > > > > > I would highly recommend to check indexes and how the updates
>> > work
>> > > in
>> > > > > > data
>> > > > > > > index to avoid full scan table and lock the full table. Some
>> db
>> > are
>> > > > > very
>> > > > > > > sensitive to that.
>> > > > > > >
>> > > > > > > El mié, 7 feb 2024, 18:41, Francisco Javier Tirado Sarti <
>> > > > > > > ftirados@redhat.com> escribió:
>> > > > > > >
>> > > > > > > > Hi Martin,
>> > > > > > > > While I analyze the data, let me ask you if it is possible
>> to
>> > > > perform
>> > > > > > > > another check (similar in a way to disabling data-index like
>> > you
>> > > > do)
>> > > > > > Can
>> > > > > > > > you switch to MongoDB persistence and check if the same
>> > > degradation
>> > > > > > that
>> > > > > > > is
>> > > > > > > > there for postgres remains?
>> > > > > > > > I do not know if this is feasible but will certainly
>> indicate
>> > the
>> > > > > > problem
>> > > > > > > > is on the postgres storage layer and I do not have a clear
>> > > > prediction
>> > > > > > of
>> > > > > > > > what we will see when doing this switch.
>> > > > > > > >
>> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler
>> > > > <mweiler@ibm.com.invalid
>> > > > > >
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi Francisco,
>> > > > > > > > >
>> > > > > > > > > thanks for your work on this important topic!
>> > > > > > > > >
>> > > > > > > > > I would like to share some test results here, which might
>> > help
>> > > to
>> > > > > > > improve
>> > > > > > > > > the codebase even further. I am using the jmeter based
>> test
>> > > case
>> > > > > from
>> > > > > > > > Pere
>> > > > > > > > > and Enrique (thanks guys!) [1] which uses a load of 30
>> > threads
>> > > to
>> > > > > > > > >
>> > > > > > > > > 1) start a new process instance (POST)
>> > > > > > > > > 2) retrieve tasks for a user (GET)
>> > > > > > > > > 3) fetches task details (GET)
>> > > > > > > > > 4) complete a task (POST)
>> > > > > > > > > 5) execute a query on data-audit
>> > > > > > > > >
>> > > > > > > > > With this test setup, I noticed that the performance for
>> the
>> > > POST
>> > > > > > > > > requests, in particular the one to start a new process
>> > > instance,
>> > > > > > > degrades
>> > > > > > > > > over time - see graph [2]. If I run the same test without
>> > > > > data-index,
>> > > > > > > > then
>> > > > > > > > > there is no such performance degradation [3]. You can
>> find a
>> > > > thread
>> > > > > > > dump
>> > > > > > > > > captured a few minutes into the first test here [4] that
>> > might
>> > > > help
>> > > > > > to
>> > > > > > > > see
>> > > > > > > > > some of the contention points.
>> > > > > > > > >
>> > > > > > > > > I'd appreciate if you could take a look and see if there
>> is
>> > > > > something
>> > > > > > > > that
>> > > > > > > > > can be further improved based on your previous work. If
>> you
>> > > need
>> > > > > any
>> > > > > > > > > additional data, let me know, but otherwise it is
>> > > straightforward
>> > > > > to
>> > > > > > > run
>> > > > > > > > > the jmeter test as well.
>> > > > > > > > >
>> > > > > > > > > Thanks,
>> > > > > > > > > Martin
>> > > > > > > > >
>> > > > > > > > > [1]
>> https://github.com/pefernan/job-service-refactor-test/
>> > > > > > > > > [2]
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
>> > > > > > > > > [3]
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
>> > > > > > > > > [4]
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
>> > > > > > > > >
>> > > > > > > > > ________________________________________
>> > > > > > > > > From: Francisco Javier Tirado Sarti <ft...@redhat.com>
>> > > > > > > > > Sent: Wednesday, January 17, 2024 9:13 AM
>> > > > > > > > > To: dev@kie.apache.org
>> > > > > > > > > Cc: Pere Fernandez Perez
>> > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues
>> with
>> > > > > > data-index
>> > > > > > > > > persistence addon
>> > > > > > > > >
>> > > > > > > > > Hi Alex,
>> > > > > > > > > I did not take times (which depends on a number of
>> variables
>> > > that
>> > > > > > > > > drastically change between environments), but verify that
>> the
>> > > > > number
>> > > > > > of
>> > > > > > > > > updates has been reduced drastically without losing
>> > > > functionality,
>> > > > > > > which
>> > > > > > > > is
>> > > > > > > > > objectively a good thing. If before the change, for every
>> > node
>> > > > > > > executed,
>> > > > > > > > we
>> > > > > > > > > have an update for every node previously executed, so if a
>> > > > process
>> > > > > > have
>> > > > > > > > 50
>> > > > > > > > > nodes to execute, we were performing nearly 50*51/2
>> updates,
>> > > > which
>> > > > > > > gives
>> > > > > > > > us
>> > > > > > > > > a total of  1275 updates, now we have just one for every
>> node
>> > > > being
>> > > > > > > > > executed, implying a total of 50 updates.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli <
>> > > alex@porcelli.me>
>> > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Francisco,
>> > > > > > > > > >
>> > > > > > > > > > I noticed that your PR has been merged, but I was
>> expecting
>> > > (at
>> > > > > > least
>> > > > > > > > > > was my understanding from this thread) that before
>> merging
>> > > some
>> > > > > > > > > > benchmark data would be shared in advance - to assess
>> the
>> > > > > > > cost/benefit
>> > > > > > > > > > of such a decent size change.
>> > > > > > > > > >
>> > > > > > > > > > Do you have any information to share?
>> > > > > > > > > >
>> > > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM Francisco Javier Tirado
>> > Sarti
>> > > > > > > > > > <ft...@redhat.com> wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > Yes, as intended, now we have one select and one
>> > > > insert/update
>> > > > > > per
>> > > > > > > > node
>> > > > > > > > > > > event.
>> > > > > > > > > > > I moved the PR as ready for review and give @Pere
>> > Fernandez
>> > > > > Perez
>> > > > > > > > > > > <pe...@redhat.com> permission to the branch so he
>> can
>> > > > edit
>> > > > > it
>> > > > > > > in
>> > > > > > > > > the
>> > > > > > > > > > > next two weeks (Ill be on PTO)  if desired, before
>> > merging.
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex Porcelli <
>> > > > > alex@porcelli.me>
>> > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Cool, thank you Francisco!
>> > > > > > > > > > > >
>> > > > > > > > > > > > Did you manage to get some preliminary data about
>> > > > > improvements?
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM Francisco Javier
>> > Tirado
>> > > > > Sarti
>> > > > > > > > > > > > <ft...@redhat.com> wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Yes, after some delay because of quarkus 3
>> migration.
>> > > Im
>> > > > > > > refining
>> > > > > > > > > > this
>> > > > > > > > > > > > > draft PR
>> > > > > > > > > > > > >
>> > > > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex Porcelli <
>> > > > > > > alex@porcelli.me>
>> > > > > > > > > > wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > Any update or new findings on this topic?
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM Francisco Javier
>> > > Tirado
>> > > > > > Sarti
>> > > > > > > > > > > > > > <ft...@redhat.com> wrote:
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Hi Alex,
>> > > > > > > > > > > > > > > After considering different options to improve
>> > > > > > performance,
>> > > > > > > > we
>> > > > > > > > > > feel
>> > > > > > > > > > > > that
>> > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > is time to "partially" move away from the
>> current
>> > > Map
>> > > > > > style
>> > > > > > > > > > > > interface (
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
>> > > > > > > > > > > > > > )
>> > > > > > > > > > > > > > > which was shared with Trusty, to one more
>> > suitable
>> > > > for
>> > > > > > > usage
>> > > > > > > > > > with a
>> > > > > > > > > > > > > > > relational DB like postgresql (but still
>> > compatible
>> > > > > with
>> > > > > > > big
>> > > > > > > > > > table
>> > > > > > > > > > > > dbs).
>> > > > > > > > > > > > > > > The idea will be to replace generic Storage
>> > > interface
>> > > > > by
>> > > > > > > four
>> > > > > > > > > > > > specific
>> > > > > > > > > > > > > > > interfaces (which will inherit from a common
>> one
>> > > that
>> > > > > > keeps
>> > > > > > > > the
>> > > > > > > > > > query
>> > > > > > > > > > > > > > part
>> > > > > > > > > > > > > > > at is it. with get and query methods), that
>> will
>> > > > > include
>> > > > > > > the
>> > > > > > > > > > required
>> > > > > > > > > > > > > > > modification operations for the four DataIndex
>> > > > > "domains":
>> > > > > > > > > > > > > > processinstance,
>> > > > > > > > > > > > > > > usertask, processdefinitions and jobs. Those
>> > > > interfaces
>> > > > > > > will
>> > > > > > > > > > define
>> > > > > > > > > > > > > > methods
>> > > > > > > > > > > > > > > like addNode, addVariable, updateTask,
>> > > > > addAttachment.....
>> > > > > > > > that
>> > > > > > > > > > will
>> > > > > > > > > > > > allow
>> > > > > > > > > > > > > > > the persistent layer implementation  to just
>> > update
>> > > > the
>> > > > > > > > needed
>> > > > > > > > > > info
>> > > > > > > > > > > > in
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > DB  (for example, for addNode in Postgres,
>> just
>> > > > insert
>> > > > > a
>> > > > > > > row
>> > > > > > > > > into
>> > > > > > > > > > > > nodes
>> > > > > > > > > > > > > > > table, for addNode in Mongo, basically the
>> same
>> > > > atomic
>> > > > > > > upsert
>> > > > > > > > > > > > operation
>> > > > > > > > > > > > > > > that is currently done). Therefore, we
>> increase
>> > > > > > performance
>> > > > > > > > for
>> > > > > > > > > > > > Postgres
>> > > > > > > > > > > > > > > and keep the current one for Mongo. The
>> current
>> > DB
>> > > > > > schemas
>> > > > > > > > > won't
>> > > > > > > > > > be
>> > > > > > > > > > > > > > > touched.
>> > > > > > > > > > > > > > > Since the code change is large, I do not think
>> > I'll
>> > > > be
>> > > > > > able
>> > > > > > > > to
>> > > > > > > > > > have
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > > PR
>> > > > > > > > > > > > > > > ready till next week.
>> > > > > > > > > > > > > > > But before starting, please let me know if
>> that
>> > > > > approach
>> > > > > > is
>> > > > > > > > > fine
>> > > > > > > > > > for
>> > > > > > > > > > > > you.
>> > > > > > > > > > > > > > > Best regards.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 6:55 PM Alex Porcelli
>> <
>> > > > > > > > > alex@porcelli.me>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Thank you Francisco to getting deeper on
>> this…
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Looking forward to see the results of your
>> > > > suggested
>> > > > > > > > > > improvements.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 9:40 AM Francisco
>> > Javier
>> > > > > Tirado
>> > > > > > > > > Sarti <
>> > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > I forgot to attach the queries
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 3:04 PM Francisco
>> > > Javier
>> > > > > > Tirado
>> > > > > > > > > > Sarti <
>> > > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >> Hi,
>> > > > > > > > > > > > > > > > >> A brief update on this topic.
>> > > > > > > > > > > > > > > > >> After doing a simple test with example
>> > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
>> > > > > > > > > > > > > > > > ,
>> > > > > > > > > > > > > > > > >> the number of updates over Nodes table is
>> > n*n,
>> > > > so
>> > > > > we
>> > > > > > > > > manage
>> > > > > > > > > > to
>> > > > > > > > > > > > > > obtain a
>> > > > > > > > > > > > > > > > >> perfect quadratic performance
>> degradation.
>> > The
>> > > > > > problem
>> > > > > > > > is
>> > > > > > > > > > worse
>> > > > > > > > > > > > in
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > case
>> > > > > > > > > > > > > > > > >> of Serverless Workflow than in BPMN
>> because
>> > we
>> > > > the
>> > > > > > > > number
>> > > > > > > > > of
>> > > > > > > > > > > > nodes
>> > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > >> greater than the number of states. In
>> that
>> > > > > example N
>> > > > > > > is
>> > > > > > > > > 16,
>> > > > > > > > > > but
>> > > > > > > > > > > > for
>> > > > > > > > > > > > > > a
>> > > > > > > > > > > > > > > > more
>> > > > > > > > > > > > > > > > >> complex workflow it would be certainly
>> > large.
>> > > > > > > > > > > > > > > > >> I think that this is more related to how
>> we
>> > > are
>> > > > > > > handling
>> > > > > > > > > > JPA in
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > code,
>> > > > > > > > > > > > > > > > >> in particular the mapping from model to
>> > entity
>> > > > > > > > (basically
>> > > > > > > > > > JPA is
>> > > > > > > > > > > > > > blind
>> > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > >> has to update all nodes for every write
>> > > because
>> > > > it
>> > > > > > > > > believes
>> > > > > > > > > > the
>> > > > > > > > > > > > > > node has
>> > > > > > > > > > > > > > > > >> been updated, although it is not) than an
>> > > issue
>> > > > in
>> > > > > > the
>> > > > > > > > > table
>> > > > > > > > > > > > > > definition.
>> > > > > > > > > > > > > > > > >> In fact, when using JPA, separating the
>> > server
>> > > > > model
>> > > > > > > > from
>> > > > > > > > > > the
>> > > > > > > > > > > > JPA
>> > > > > > > > > > > > > > > > entity is
>> > > > > > > > > > > > > > > > >> not a good idea, especially if the entity
>> > > > contains
>> > > > > > > > > > collections.
>> > > > > > > > > > > > I
>> > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > > try
>> > > > > > > > > > > > > > > > >> to change that without breaking anything.
>> > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > >> On Wed, Nov 22, 2023 at 12:10 PM Enrique
>> > > > Gonzalez
>> > > > > > > > > Martinez <
>> > > > > > > > > > > > > > > > >> egonzalez@apache.org> wrote:
>> > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > >>> After the events split you now will
>> need to
>> > > > > create
>> > > > > > a
>> > > > > > > > node
>> > > > > > > > > > > > instance
>> > > > > > > > > > > > > > > > >>> model instance of making independent
>> from
>> > the
>> > > > > > process
>> > > > > > > > > > instance.
>> > > > > > > > > > > > > > > > >>> That should do the trick.
>> > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > >>> Regarding deleting/inserting it was
>> fixed
>> > at
>> > > > some
>> > > > > > > > point.
>> > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > >>> El mar, 21 nov 2023 a las 20:22,
>> Francisco
>> > > > Javier
>> > > > > > > > Tirado
>> > > > > > > > > > Sarti
>> > > > > > > > > > > > > > > > >>> (<ft...@redhat.com>) escribió:
>> > > > > > > > > > > > > > > > >>> >
>> > > > > > > > > > > > > > > > >>> > Hi Martin,
>> > > > > > > > > > > > > > > > >>> > I have a task to review performance of
>> > > > > > > > > > > > > > > > >>> >
>> > > > > > > > > > > > > > > > >>> > ProcessInstanceNodeDataEventMerger
>> > > > > > > > > > > > > > > > >>> > My idea is to reduce the number of
>> delete
>> > > > > inserts
>> > > > > > > > when
>> > > > > > > > > > > > processing
>> > > > > > > > > > > > > > > > >>> events
>> > > > > > > > > > > > > > > > >>> > and try to do it incremental.
>> > > > > > > > > > > > > > > > >>> > That should improve performance.
>> > > > > > > > > > > > > > > > >>> > PS:
>> > > > > > > > > > > > > > > > >>> > I was planning to send an e-mail
>> tomorrow
>> > > > > > > announcing
>> > > > > > > > > > that in
>> > > > > > > > > > > > > > case you
>> > > > > > > > > > > > > > > > >>> were
>> > > > > > > > > > > > > > > > >>> > already working on a fix for that. I
>> > assume
>> > > > you
>> > > > > > are
>> > > > > > > > not
>> > > > > > > > > > and I
>> > > > > > > > > > > > > > would
>> > > > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > >>> > sending a PR soon.
>> > > > > > > > > > > > > > > > >>> >
>> > > > > > > > > > > > > > > > >>> > On Tue, Nov 21, 2023 at 6:09 PM Martin
>> > > Weiler
>> > > > > > > > > > > > > > > > <mweiler@ibm.com.invalid
>> > > > > > > > > > > > > > > > >>> >
>> > > > > > > > > > > > > > > > >>> > wrote:
>> > > > > > > > > > > > > > > > >>> >
>> > > > > > > > > > > > > > > > >>> > > I looked into the new examples using
>> > > > > data-index
>> > > > > > > > > > persistence
>> > > > > > > > > > > > > > addon -
>> > > > > > > > > > > > > > > > >>> Neus'
>> > > > > > > > > > > > > > > > >>> > > PR#1813 [1] for serverless and
>> Pere's
>> > > > branch
>> > > > > > [2]
>> > > > > > > > for
>> > > > > > > > > > > > workflow
>> > > > > > > > > > > > > > > > (great
>> > > > > > > > > > > > > > > > >>> job
>> > > > > > > > > > > > > > > > >>> > > both!) - and they work without
>> issues
>> > > using
>> > > > > > > single
>> > > > > > > > > > > > requests.
>> > > > > > > > > > > > > > > > >>> However, under
>> > > > > > > > > > > > > > > > >>> > > some load (I used 'ab' for testing
>> > with a
>> > > > > light
>> > > > > > > > > > > > concurrency of
>> > > > > > > > > > > > > > 10
>> > > > > > > > > > > > > > > > >>> parallel
>> > > > > > > > > > > > > > > > >>> > > requests) I ran into the following
>> > > > problems:
>> > > > > > > > > > > > > > > > >>> > >
>> > > > > > > > > > > > > > > > >>> > > (1) Large number of insert/delete
>> calls
>> > > > (eg.
>> > > > > > for
>> > > > > > > > > tables
>> > > > > > > > > > > > such as
>> > > > > > > > > > > > > > > > >>> nodes,
>> > > > > > > > > > > > > > > > >>> > > definitions, etc)
>> > > > > > > > > > > > > > > > >>> > >
>> > > > > > > > > > > > > > > > >>> > > (2) Hibernate
>> OptimisticLockExceptions
>> > /
>> > > > > > > > > > > > StaleStateExceptions
>> > > > > > > > > > > > > > > > >>> > >
>> > > > > > > > > > > > > > > > >>> > > (3) DB deadlocks
>> > > > > > > > > > > > > > > > >>> > >
>> > > > > > > > > > > > > > > > >>> > > (4) Error responses, slow response
>> > times
>> > > > > > > > > > > > > > > > >>> > >
>> > > > > > > > > > > > > > > > >>> > > The reason I am reaching out with
>> this
>> > > > topic
>> > > > > > here
>> > > > > > > > is
>> > > > > > > > > to
>> > > > > > > > > > > > find
>> > > > > > > > > > > > > > out if
>> > > > > > > > > > > > > > > > >>> we are
>> > > > > > > > > > > > > > > > >>> > > aware of this issue, and if someone
>> is
>> > > > > already
>> > > > > > > > > looking
>> > > > > > > > > > > > into or
>> > > > > > > > > > > > > > > > being
>> > > > > > > > > > > > > > > > >>> > > assigned to it?
>> > > > > > > > > > > > > > > > >>> > >
>> > > > > > > > > > > > > > > > >>> > > Thanks,
>> > > > > > > > > > > > > > > > >>> > > Martin
>> > > > > > > > > > > > > > > > >>> > >
>> > > > > > > > > > > > > > > > >>> > > [1]
>> > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > >
>> > > > > > >
>> > https://github.com/apache/incubator-kie-kogito-examples/pull/1813
>> > > > > > > > > > > > > > > > >>> > > [2]
>> > > > > > > > > > > > > > > > >>> > >
>> > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
>> > > > > > > > > > > > > > > > >>> > >
>> > > > > > > > > > > > > > > > >>> > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > >
>> > > >
>> ---------------------------------------------------------------------
>> > > > > > > > > > > > > > > > >>> > > To unsubscribe, e-mail:
>> > > > > > > > > dev-unsubscribe@kie.apache.org
>> > > > > > > > > > > > > > > > >>> > > For additional commands, e-mail:
>> > > > > > > > > > dev-help@kie.apache.org
>> > > > > > > > > > > > > > > > >>> > >
>> > > > > > > > > > > > > > > > >>> > >
>> > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > >
>> > > > > > > > > >
>> > > > > >
>> > ---------------------------------------------------------------------
>> > > > > > > > > > > > > > > > >>> To unsubscribe, e-mail:
>> > > > > > > dev-unsubscribe@kie.apache.org
>> > > > > > > > > > > > > > > > >>> For additional commands, e-mail:
>> > > > > > > > dev-help@kie.apache.org
>> > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > >
>> > > >
>> ---------------------------------------------------------------------
>> > > > > > > > > > > > > > > > > To unsubscribe, e-mail:
>> > > > > > dev-unsubscribe@kie.apache.org
>> > > > > > > > > > > > > > > > > For additional commands, e-mail:
>> > > > > > > dev-help@kie.apache.org
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > >
>> > > > > >
>> > ---------------------------------------------------------------------
>> > > > > > > > > > > > > > To unsubscribe, e-mail:
>> > > dev-unsubscribe@kie.apache.org
>> > > > > > > > > > > > > > For additional commands, e-mail:
>> > > > dev-help@kie.apache.org
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > >
>> > > >
>> ---------------------------------------------------------------------
>> > > > > > > > > > > > To unsubscribe, e-mail:
>> dev-unsubscribe@kie.apache.org
>> > > > > > > > > > > > For additional commands, e-mail:
>> > dev-help@kie.apache.org
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > >
>> > ---------------------------------------------------------------------
>> > > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
>> > > > > > > > > > For additional commands, e-mail:
>> dev-help@kie.apache.org
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > >
>> ---------------------------------------------------------------------
>> > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
>> > > > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
>> For additional commands, e-mail: dev-help@kie.apache.org
>>
>>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Francisco Javier Tirado Sarti <ft...@redhat.com>]

Hi Martin,
Good catch!. Looks like the skipping performed for process instances is not
applied to node instances. Something we definitely need to review on the
runtimes side.

On Mon, Feb 12, 2024 at 11:59 PM Martin Weiler <mw...@ibm.com.invalid>
wrote:

> On a somewhat related note, testing a simple workflow (start -> script
> node -> end), I see the following messages in the logs:
> 2024-02-12 22:49:50,493 28758dde544c WARN
> [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1]
> (executor-thread-3) Skipping create of process instance id:
> 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2
>
> So far, so good. And I'd expect to see no trace of this process in the
> database if I don't have data audit enabled.
>
> However, the 'processes' table contains a row with state=2, with related
> entries in the 'nodes' table. In a load test, I see these tables grow
> significantly over time. Am I missing something to have these entries
> cleaned up automatically?
>
> ________________________________________
> From: Martin Weiler <mw...@ibm.com.INVALID>
> Sent: Monday, February 12, 2024 3:40 PM
> To: dev@kie.apache.org
> Subject: [EXTERNAL] RE: [DISCUSSION] Performance issues with data-index
> persistence addon
>
> Thanks everyone for your input. Based on this discussion, I opened the
> following PR:
> https://github.com/apache/incubator-kie-kogito-apps/pull/1985
>
> With this change, the performance seems to be stable over time:
>
> https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing
>
> Martin
>
> ________________________________________
> From: Gonzalo Muñoz <gm...@apache.org>
> Sent: Friday, February 9, 2024 9:42 AM
> To: dev@kie.apache.org
> Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with data-index
> persistence addon
>
> Great work Francisco,
> Martin, take a look at this link with some related tips (in case you find
> it useful): https://www.cybertec-postgresql.com/en/index-your-foreign-key/
>
> El vie, 9 feb 2024 a las 17:20, Francisco Javier Tirado Sarti (<
> ftirados@redhat.com>) escribió:
>
> > For the moment being, we will keep JPA till we exhaust all possibilities,
> > let's call switching from jpa to jdbc our hidden plan B ;)
> > I already told Martin, but in order everyone to know, just after writing
> > the previous email, I thought "what if Postgres is not automatically
> > indexing foreign keys like mysql?" and, eureka
> > Postgres doc
> > https://www.postgresql.org/docs/current/ddl-constraints.html
> > Mysql doc
> > https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> > These are the relevant excerpt
> >
> > *Postgresql*
> > *A foreign key must reference columns that either are a primary key or
> form
> > a unique constraint, or are columns from a non-partial unique index. This
> > means that the referenced columns always have an index to allow efficient
> > lookups on whether a referencing row has a match. Since a DELETE of a row
> > from the referenced table or an UPDATE of a referenced column will
> require
> > a scan of the referencing table for rows matching the old value, it is
> > often a good idea to index the referencing columns too. Because this is
> not
> > always needed, and there are many choices available on how to index, the
> > declaration of a foreign key constraint does not automatically create an
> > index on the referencing columns.*
> > *Mysql*
> > *MySQL requires that foreign key columns be indexed; if you create a
> table
> > with a foreign key constraint but no index on a given column, an index is
> > created. *
> >
> > So I asked Martin to especially create an index for process_instance_id
> > column on nodes table
> > I think that will fix the problem detected on the thread dump.
> > The simpler process test to verify queries are fine still stands, though
> ;)
> >
> >
> > On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <tz...@apache.org>
> wrote:
> >
> > > I always preferred pure JDBC over Hibernate myself, just for the sake
> of
> > > control of what is happening :) So I would not -1 that myself.
> > >
> > > Tibor
> > >
> > > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado Sarti <
> > > ftirados@redhat.com>
> > > napísal(a):
> > >
> > > > Hi,
> > > > Usually I do not want to talk about work in progress because
> > preliminary
> > > > conclusions are pretty volatile but, well, there are a couple of
> things
> > > > that can be concluded from the really valuable information that
> Martin
> > > > provided.
> > > > 1) In order to be able to determine if the number of statements is
> > larger
> > > > than expected, I asked Martin to test with a simpler process
> > definition.
> > > > One with just three nodes: start, script and end. The script one
> should
> > > > change just one variable. This way we can analyze if the number of
> > > queries
> > > > is the expected one. From the single log (audit was activated them)
> my
> > > > conclusion is that the number of insert/updates over processes and
> > nodes
> > > > (there a lot over task, that I will prefer to skip for now, baby
> steps)
> > > is
> > > > the expected one.
> > > > 2) Analysing the thread dump, we see around 15 threads executing this
> > > line
> > > > at
> > > >
> > > >
> > >
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > > so its pretty clear the code to be optimized ;). I'm evaluating
> > > > possibilities within JPA/Hibernate, but I'm starting to think that it
> > > might
> > > > be better to switch to JDBC and skip hibernate. Our lives will be
> > > simpler,
> > > > especially with a schema relatively simple like ours (that will be my
> > > > recommendation if I was an external consultant)
> > > >
> > > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <tz...@apache.org>
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > this will be a bit off-topic. However as far as performance, I
> think
> > we
> > > > > should think about that we have string primary keys (IDs). I would
> > > expect
> > > > > the database systems are much better with indexing numeric keys
> than
> > > > > strings. I remember from the past, when I was working with DBs,
> that
> > > > using
> > > > > strings as keys or indexes was a discouraged practice.
> > > > >
> > > > > Best regards,
> > > > > Tibor
> > > > >
> > > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler <mw...@ibm.com.invalid>
> > > > > napísal(a):
> > > > >
> > > > > > I changed the test to use MongoDB [1] and I don't see a
> performance
> > > > > > degradation with this setup [2].
> > > > > >
> > > > > > Please keep us posted of your findings. Thanks!
> > > > > >
> > > > > > Martin
> > > > > >
> > > > > > [1]
> > > > >
> > https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > > > [2]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > > >
> > > > > > ________________________________________
> > > > > > From: Francisco Javier Tirado Sarti <ft...@redhat.com>
> > > > > > Sent: Wednesday, February 7, 2024 11:40 AM
> > > > > > To: dev@kie.apache.org
> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > > data-index
> > > > > > persistence addon
> > > > > >
> > > > > > yes, it can be index degradation because of size, but I believe
> (I
> > > > might
> > > > > be
> > > > > > wrong) the db is too small (yet) for that.
> > > > > > But, eventually, Postgres, when the DB is huge enough,
> unavoidably
> > > will
> > > > > > behave like the graphic that Martin sent.
> > > > > > Since I believe we are not huge enough (yet), lets rule out
> another
> > > > issue
> > > > > > by analysing the sql logs (I requested those to Martin offline
> and
> > he
> > > > is
> > > > > > going to kindly collect them).
> > > > > > Also Im curious to know if Mongo behave in the same way.
> > > > > >
> > > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez Martinez <
> > > > > > egonzalez@apache.org> wrote:
> > > > > >
> > > > > > > Hi Francisco,
> > > > > > > I would highly recommend to check indexes and how the updates
> > work
> > > in
> > > > > > data
> > > > > > > index to avoid full scan table and lock the full table. Some db
> > are
> > > > > very
> > > > > > > sensitive to that.
> > > > > > >
> > > > > > > El mié, 7 feb 2024, 18:41, Francisco Javier Tirado Sarti <
> > > > > > > ftirados@redhat.com> escribió:
> > > > > > >
> > > > > > > > Hi Martin,
> > > > > > > > While I analyze the data, let me ask you if it is possible to
> > > > perform
> > > > > > > > another check (similar in a way to disabling data-index like
> > you
> > > > do)
> > > > > > Can
> > > > > > > > you switch to MongoDB persistence and check if the same
> > > degradation
> > > > > > that
> > > > > > > is
> > > > > > > > there for postgres remains?
> > > > > > > > I do not know if this is feasible but will certainly indicate
> > the
> > > > > > problem
> > > > > > > > is on the postgres storage layer and I do not have a clear
> > > > prediction
> > > > > > of
> > > > > > > > what we will see when doing this switch.
> > > > > > > >
> > > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler
> > > > <mweiler@ibm.com.invalid
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Francisco,
> > > > > > > > >
> > > > > > > > > thanks for your work on this important topic!
> > > > > > > > >
> > > > > > > > > I would like to share some test results here, which might
> > help
> > > to
> > > > > > > improve
> > > > > > > > > the codebase even further. I am using the jmeter based test
> > > case
> > > > > from
> > > > > > > > Pere
> > > > > > > > > and Enrique (thanks guys!) [1] which uses a load of 30
> > threads
> > > to
> > > > > > > > >
> > > > > > > > > 1) start a new process instance (POST)
> > > > > > > > > 2) retrieve tasks for a user (GET)
> > > > > > > > > 3) fetches task details (GET)
> > > > > > > > > 4) complete a task (POST)
> > > > > > > > > 5) execute a query on data-audit
> > > > > > > > >
> > > > > > > > > With this test setup, I noticed that the performance for
> the
> > > POST
> > > > > > > > > requests, in particular the one to start a new process
> > > instance,
> > > > > > > degrades
> > > > > > > > > over time - see graph [2]. If I run the same test without
> > > > > data-index,
> > > > > > > > then
> > > > > > > > > there is no such performance degradation [3]. You can find
> a
> > > > thread
> > > > > > > dump
> > > > > > > > > captured a few minutes into the first test here [4] that
> > might
> > > > help
> > > > > > to
> > > > > > > > see
> > > > > > > > > some of the contention points.
> > > > > > > > >
> > > > > > > > > I'd appreciate if you could take a look and see if there is
> > > > > something
> > > > > > > > that
> > > > > > > > > can be further improved based on your previous work. If you
> > > need
> > > > > any
> > > > > > > > > additional data, let me know, but otherwise it is
> > > straightforward
> > > > > to
> > > > > > > run
> > > > > > > > > the jmeter test as well.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Martin
> > > > > > > > >
> > > > > > > > > [1] https://github.com/pefernan/job-service-refactor-test/
> > > > > > > > > [2]
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > > > > > [3]
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > > > > > [4]
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > > > > >
> > > > > > > > > ________________________________________
> > > > > > > > > From: Francisco Javier Tirado Sarti <ft...@redhat.com>
> > > > > > > > > Sent: Wednesday, January 17, 2024 9:13 AM
> > > > > > > > > To: dev@kie.apache.org
> > > > > > > > > Cc: Pere Fernandez Perez
> > > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues
> with
> > > > > > data-index
> > > > > > > > > persistence addon
> > > > > > > > >
> > > > > > > > > Hi Alex,
> > > > > > > > > I did not take times (which depends on a number of
> variables
> > > that
> > > > > > > > > drastically change between environments), but verify that
> the
> > > > > number
> > > > > > of
> > > > > > > > > updates has been reduced drastically without losing
> > > > functionality,
> > > > > > > which
> > > > > > > > is
> > > > > > > > > objectively a good thing. If before the change, for every
> > node
> > > > > > > executed,
> > > > > > > > we
> > > > > > > > > have an update for every node previously executed, so if a
> > > > process
> > > > > > have
> > > > > > > > 50
> > > > > > > > > nodes to execute, we were performing nearly 50*51/2
> updates,
> > > > which
> > > > > > > gives
> > > > > > > > us
> > > > > > > > > a total of  1275 updates, now we have just one for every
> node
> > > > being
> > > > > > > > > executed, implying a total of 50 updates.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli <
> > > alex@porcelli.me>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Francisco,
> > > > > > > > > >
> > > > > > > > > > I noticed that your PR has been merged, but I was
> expecting
> > > (at
> > > > > > least
> > > > > > > > > > was my understanding from this thread) that before
> merging
> > > some
> > > > > > > > > > benchmark data would be shared in advance - to assess the
> > > > > > > cost/benefit
> > > > > > > > > > of such a decent size change.
> > > > > > > > > >
> > > > > > > > > > Do you have any information to share?
> > > > > > > > > >
> > > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM Francisco Javier Tirado
> > Sarti
> > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Yes, as intended, now we have one select and one
> > > > insert/update
> > > > > > per
> > > > > > > > node
> > > > > > > > > > > event.
> > > > > > > > > > > I moved the PR as ready for review and give @Pere
> > Fernandez
> > > > > Perez
> > > > > > > > > > > <pe...@redhat.com> permission to the branch so he
> can
> > > > edit
> > > > > it
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > next two weeks (Ill be on PTO)  if desired, before
> > merging.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex Porcelli <
> > > > > alex@porcelli.me>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Cool, thank you Francisco!
> > > > > > > > > > > >
> > > > > > > > > > > > Did you manage to get some preliminary data about
> > > > > improvements?
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM Francisco Javier
> > Tirado
> > > > > Sarti
> > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yes, after some delay because of quarkus 3
> migration.
> > > Im
> > > > > > > refining
> > > > > > > > > > this
> > > > > > > > > > > > > draft PR
> > > > > > > > > > > > >
> > > > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex Porcelli <
> > > > > > > alex@porcelli.me>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Any update or new findings on this topic?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM Francisco Javier
> > > Tirado
> > > > > > Sarti
> > > > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > > > > > After considering different options to improve
> > > > > > performance,
> > > > > > > > we
> > > > > > > > > > feel
> > > > > > > > > > > > that
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > is time to "partially" move away from the
> current
> > > Map
> > > > > > style
> > > > > > > > > > > > interface (
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > which was shared with Trusty, to one more
> > suitable
> > > > for
> > > > > > > usage
> > > > > > > > > > with a
> > > > > > > > > > > > > > > relational DB like postgresql (but still
> > compatible
> > > > > with
> > > > > > > big
> > > > > > > > > > table
> > > > > > > > > > > > dbs).
> > > > > > > > > > > > > > > The idea will be to replace generic Storage
> > > interface
> > > > > by
> > > > > > > four
> > > > > > > > > > > > specific
> > > > > > > > > > > > > > > interfaces (which will inherit from a common
> one
> > > that
> > > > > > keeps
> > > > > > > > the
> > > > > > > > > > query
> > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > at is it. with get and query methods), that
> will
> > > > > include
> > > > > > > the
> > > > > > > > > > required
> > > > > > > > > > > > > > > modification operations for the four DataIndex
> > > > > "domains":
> > > > > > > > > > > > > > processinstance,
> > > > > > > > > > > > > > > usertask, processdefinitions and jobs. Those
> > > > interfaces
> > > > > > > will
> > > > > > > > > > define
> > > > > > > > > > > > > > methods
> > > > > > > > > > > > > > > like addNode, addVariable, updateTask,
> > > > > addAttachment.....
> > > > > > > > that
> > > > > > > > > > will
> > > > > > > > > > > > allow
> > > > > > > > > > > > > > > the persistent layer implementation  to just
> > update
> > > > the
> > > > > > > > needed
> > > > > > > > > > info
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > DB  (for example, for addNode in Postgres, just
> > > > insert
> > > > > a
> > > > > > > row
> > > > > > > > > into
> > > > > > > > > > > > nodes
> > > > > > > > > > > > > > > table, for addNode in Mongo, basically the same
> > > > atomic
> > > > > > > upsert
> > > > > > > > > > > > operation
> > > > > > > > > > > > > > > that is currently done). Therefore, we increase
> > > > > > performance
> > > > > > > > for
> > > > > > > > > > > > Postgres
> > > > > > > > > > > > > > > and keep the current one for Mongo. The current
> > DB
> > > > > > schemas
> > > > > > > > > won't
> > > > > > > > > > be
> > > > > > > > > > > > > > > touched.
> > > > > > > > > > > > > > > Since the code change is large, I do not think
> > I'll
> > > > be
> > > > > > able
> > > > > > > > to
> > > > > > > > > > have
> > > > > > > > > > > > the
> > > > > > > > > > > > > > PR
> > > > > > > > > > > > > > > ready till next week.
> > > > > > > > > > > > > > > But before starting, please let me know if that
> > > > > approach
> > > > > > is
> > > > > > > > > fine
> > > > > > > > > > for
> > > > > > > > > > > > you.
> > > > > > > > > > > > > > > Best regards.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 6:55 PM Alex Porcelli <
> > > > > > > > > alex@porcelli.me>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thank you Francisco to getting deeper on
> this…
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Looking forward to see the results of your
> > > > suggested
> > > > > > > > > > improvements.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 9:40 AM Francisco
> > Javier
> > > > > Tirado
> > > > > > > > > Sarti <
> > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I forgot to attach the queries
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 3:04 PM Francisco
> > > Javier
> > > > > > Tirado
> > > > > > > > > > Sarti <
> > > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >> Hi,
> > > > > > > > > > > > > > > > >> A brief update on this topic.
> > > > > > > > > > > > > > > > >> After doing a simple test with example
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > > > > > > > > > > ,
> > > > > > > > > > > > > > > > >> the number of updates over Nodes table is
> > n*n,
> > > > so
> > > > > we
> > > > > > > > > manage
> > > > > > > > > > to
> > > > > > > > > > > > > > obtain a
> > > > > > > > > > > > > > > > >> perfect quadratic performance degradation.
> > The
> > > > > > problem
> > > > > > > > is
> > > > > > > > > > worse
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > >> of Serverless Workflow than in BPMN
> because
> > we
> > > > the
> > > > > > > > number
> > > > > > > > > of
> > > > > > > > > > > > nodes
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > >> greater than the number of states. In that
> > > > > example N
> > > > > > > is
> > > > > > > > > 16,
> > > > > > > > > > but
> > > > > > > > > > > > for
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > >> complex workflow it would be certainly
> > large.
> > > > > > > > > > > > > > > > >> I think that this is more related to how
> we
> > > are
> > > > > > > handling
> > > > > > > > > > JPA in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > code,
> > > > > > > > > > > > > > > > >> in particular the mapping from model to
> > entity
> > > > > > > > (basically
> > > > > > > > > > JPA is
> > > > > > > > > > > > > > blind
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > >> has to update all nodes for every write
> > > because
> > > > it
> > > > > > > > > believes
> > > > > > > > > > the
> > > > > > > > > > > > > > node has
> > > > > > > > > > > > > > > > >> been updated, although it is not) than an
> > > issue
> > > > in
> > > > > > the
> > > > > > > > > table
> > > > > > > > > > > > > > definition.
> > > > > > > > > > > > > > > > >> In fact, when using JPA, separating the
> > server
> > > > > model
> > > > > > > > from
> > > > > > > > > > the
> > > > > > > > > > > > JPA
> > > > > > > > > > > > > > > > entity is
> > > > > > > > > > > > > > > > >> not a good idea, especially if the entity
> > > > contains
> > > > > > > > > > collections.
> > > > > > > > > > > > I
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > try
> > > > > > > > > > > > > > > > >> to change that without breaking anything.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> On Wed, Nov 22, 2023 at 12:10 PM Enrique
> > > > Gonzalez
> > > > > > > > > Martinez <
> > > > > > > > > > > > > > > > >> egonzalez@apache.org> wrote:
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >>> After the events split you now will need
> to
> > > > > create
> > > > > > a
> > > > > > > > node
> > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > >>> model instance of making independent from
> > the
> > > > > > process
> > > > > > > > > > instance.
> > > > > > > > > > > > > > > > >>> That should do the trick.
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> Regarding deleting/inserting it was fixed
> > at
> > > > some
> > > > > > > > point.
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> El mar, 21 nov 2023 a las 20:22,
> Francisco
> > > > Javier
> > > > > > > > Tirado
> > > > > > > > > > Sarti
> > > > > > > > > > > > > > > > >>> (<ft...@redhat.com>) escribió:
> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > > >>> > Hi Martin,
> > > > > > > > > > > > > > > > >>> > I have a task to review performance of
> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > > >>> > ProcessInstanceNodeDataEventMerger
> > > > > > > > > > > > > > > > >>> > My idea is to reduce the number of
> delete
> > > > > inserts
> > > > > > > > when
> > > > > > > > > > > > processing
> > > > > > > > > > > > > > > > >>> events
> > > > > > > > > > > > > > > > >>> > and try to do it incremental.
> > > > > > > > > > > > > > > > >>> > That should improve performance.
> > > > > > > > > > > > > > > > >>> > PS:
> > > > > > > > > > > > > > > > >>> > I was planning to send an e-mail
> tomorrow
> > > > > > > announcing
> > > > > > > > > > that in
> > > > > > > > > > > > > > case you
> > > > > > > > > > > > > > > > >>> were
> > > > > > > > > > > > > > > > >>> > already working on a fix for that. I
> > assume
> > > > you
> > > > > > are
> > > > > > > > not
> > > > > > > > > > and I
> > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > >>> > sending a PR soon.
> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > > >>> > On Tue, Nov 21, 2023 at 6:09 PM Martin
> > > Weiler
> > > > > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > > >>> > wrote:
> > > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > > >>> > > I looked into the new examples using
> > > > > data-index
> > > > > > > > > > persistence
> > > > > > > > > > > > > > addon -
> > > > > > > > > > > > > > > > >>> Neus'
> > > > > > > > > > > > > > > > >>> > > PR#1813 [1] for serverless and Pere's
> > > > branch
> > > > > > [2]
> > > > > > > > for
> > > > > > > > > > > > workflow
> > > > > > > > > > > > > > > > (great
> > > > > > > > > > > > > > > > >>> job
> > > > > > > > > > > > > > > > >>> > > both!) - and they work without issues
> > > using
> > > > > > > single
> > > > > > > > > > > > requests.
> > > > > > > > > > > > > > > > >>> However, under
> > > > > > > > > > > > > > > > >>> > > some load (I used 'ab' for testing
> > with a
> > > > > light
> > > > > > > > > > > > concurrency of
> > > > > > > > > > > > > > 10
> > > > > > > > > > > > > > > > >>> parallel
> > > > > > > > > > > > > > > > >>> > > requests) I ran into the following
> > > > problems:
> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > > >>> > > (1) Large number of insert/delete
> calls
> > > > (eg.
> > > > > > for
> > > > > > > > > tables
> > > > > > > > > > > > such as
> > > > > > > > > > > > > > > > >>> nodes,
> > > > > > > > > > > > > > > > >>> > > definitions, etc)
> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > > >>> > > (2) Hibernate
> OptimisticLockExceptions
> > /
> > > > > > > > > > > > StaleStateExceptions
> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > > >>> > > (3) DB deadlocks
> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > > >>> > > (4) Error responses, slow response
> > times
> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > > >>> > > The reason I am reaching out with
> this
> > > > topic
> > > > > > here
> > > > > > > > is
> > > > > > > > > to
> > > > > > > > > > > > find
> > > > > > > > > > > > > > out if
> > > > > > > > > > > > > > > > >>> we are
> > > > > > > > > > > > > > > > >>> > > aware of this issue, and if someone
> is
> > > > > already
> > > > > > > > > looking
> > > > > > > > > > > > into or
> > > > > > > > > > > > > > > > being
> > > > > > > > > > > > > > > > >>> > > assigned to it?
> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > > >>> > > Thanks,
> > > > > > > > > > > > > > > > >>> > > Martin
> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > > >>> > > [1]
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > >
> > > > > > >
> > https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > > > > > > > > > > > > >>> > > [2]
> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > > > > > > > > > >>> > > To unsubscribe, e-mail:
> > > > > > > > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > > > > > >>> > > For additional commands, e-mail:
> > > > > > > > > > dev-help@kie.apache.org
> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >
> > > > > > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > > > > > > > > > >>> To unsubscribe, e-mail:
> > > > > > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > > > > > >>> For additional commands, e-mail:
> > > > > > > > dev-help@kie.apache.org
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > > > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > > > > > > For additional commands, e-mail:
> > > > > > > dev-help@kie.apache.org
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > > > For additional commands, e-mail:
> > > > dev-help@kie.apache.org
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > > > > > To unsubscribe, e-mail:
> dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > For additional commands, e-mail:
> > dev-help@kie.apache.org
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> For additional commands, e-mail: dev-help@kie.apache.org
>
>

RE: [DISCUSSION] Performance issues with data-index persistence addon

[Martin Weiler <mw...@ibm.com.INVALID>]

On a somewhat related note, testing a simple workflow (start -> script node -> end), I see the following messages in the logs:
2024-02-12 22:49:50,493 28758dde544c WARN  [org.kie.kogito.persistence.jdbc.JDBCProcessInstances:-1] (executor-thread-3) Skipping create of process instance id: 7083088e-b899-47cb-b85c-5d9ccb0aa166, state: 2

So far, so good. And I'd expect to see no trace of this process in the database if I don't have data audit enabled.

However, the 'processes' table contains a row with state=2, with related entries in the 'nodes' table. In a load test, I see these tables grow significantly over time. Am I missing something to have these entries cleaned up automatically?

________________________________________
From: Martin Weiler <mw...@ibm.com.INVALID>
Sent: Monday, February 12, 2024 3:40 PM
To: dev@kie.apache.org
Subject: [EXTERNAL] RE: [DISCUSSION] Performance issues with data-index persistence addon

Thanks everyone for your input. Based on this discussion, I opened the following PR:
https://github.com/apache/incubator-kie-kogito-apps/pull/1985

With this change, the performance seems to be stable over time:
https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing

Martin

________________________________________
From: Gonzalo Muñoz <gm...@apache.org>
Sent: Friday, February 9, 2024 9:42 AM
To: dev@kie.apache.org
Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with data-index persistence addon

Great work Francisco,
Martin, take a look at this link with some related tips (in case you find
it useful): https://www.cybertec-postgresql.com/en/index-your-foreign-key/

El vie, 9 feb 2024 a las 17:20, Francisco Javier Tirado Sarti (<
ftirados@redhat.com>) escribió:

> For the moment being, we will keep JPA till we exhaust all possibilities,
> let's call switching from jpa to jdbc our hidden plan B ;)
> I already told Martin, but in order everyone to know, just after writing
> the previous email, I thought "what if Postgres is not automatically
> indexing foreign keys like mysql?" and, eureka
> Postgres doc
> https://www.postgresql.org/docs/current/ddl-constraints.html
> Mysql doc
> https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> These are the relevant excerpt
>
> *Postgresql*
> *A foreign key must reference columns that either are a primary key or form
> a unique constraint, or are columns from a non-partial unique index. This
> means that the referenced columns always have an index to allow efficient
> lookups on whether a referencing row has a match. Since a DELETE of a row
> from the referenced table or an UPDATE of a referenced column will require
> a scan of the referencing table for rows matching the old value, it is
> often a good idea to index the referencing columns too. Because this is not
> always needed, and there are many choices available on how to index, the
> declaration of a foreign key constraint does not automatically create an
> index on the referencing columns.*
> *Mysql*
> *MySQL requires that foreign key columns be indexed; if you create a table
> with a foreign key constraint but no index on a given column, an index is
> created. *
>
> So I asked Martin to especially create an index for process_instance_id
> column on nodes table
> I think that will fix the problem detected on the thread dump.
> The simpler process test to verify queries are fine still stands, though ;)
>
>
> On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <tz...@apache.org> wrote:
>
> > I always preferred pure JDBC over Hibernate myself, just for the sake of
> > control of what is happening :) So I would not -1 that myself.
> >
> > Tibor
> >
> > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado Sarti <
> > ftirados@redhat.com>
> > napísal(a):
> >
> > > Hi,
> > > Usually I do not want to talk about work in progress because
> preliminary
> > > conclusions are pretty volatile but, well, there are a couple of things
> > > that can be concluded from the really valuable information that Martin
> > > provided.
> > > 1) In order to be able to determine if the number of statements is
> larger
> > > than expected, I asked Martin to test with a simpler process
> definition.
> > > One with just three nodes: start, script and end. The script one should
> > > change just one variable. This way we can analyze if the number of
> > queries
> > > is the expected one. From the single log (audit was activated them) my
> > > conclusion is that the number of insert/updates over processes and
> nodes
> > > (there a lot over task, that I will prefer to skip for now, baby steps)
> > is
> > > the expected one.
> > > 2) Analysing the thread dump, we see around 15 threads executing this
> > line
> > > at
> > >
> > >
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > so its pretty clear the code to be optimized ;). I'm evaluating
> > > possibilities within JPA/Hibernate, but I'm starting to think that it
> > might
> > > be better to switch to JDBC and skip hibernate. Our lives will be
> > simpler,
> > > especially with a schema relatively simple like ours (that will be my
> > > recommendation if I was an external consultant)
> > >
> > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <tz...@apache.org>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > this will be a bit off-topic. However as far as performance, I think
> we
> > > > should think about that we have string primary keys (IDs). I would
> > expect
> > > > the database systems are much better with indexing numeric keys than
> > > > strings. I remember from the past, when I was working with DBs, that
> > > using
> > > > strings as keys or indexes was a discouraged practice.
> > > >
> > > > Best regards,
> > > > Tibor
> > > >
> > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler <mw...@ibm.com.invalid>
> > > > napísal(a):
> > > >
> > > > > I changed the test to use MongoDB [1] and I don't see a performance
> > > > > degradation with this setup [2].
> > > > >
> > > > > Please keep us posted of your findings. Thanks!
> > > > >
> > > > > Martin
> > > > >
> > > > > [1]
> > > >
> https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > > [2]
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > >
> > > > > ________________________________________
> > > > > From: Francisco Javier Tirado Sarti <ft...@redhat.com>
> > > > > Sent: Wednesday, February 7, 2024 11:40 AM
> > > > > To: dev@kie.apache.org
> > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > data-index
> > > > > persistence addon
> > > > >
> > > > > yes, it can be index degradation because of size, but I believe (I
> > > might
> > > > be
> > > > > wrong) the db is too small (yet) for that.
> > > > > But, eventually, Postgres, when the DB is huge enough, unavoidably
> > will
> > > > > behave like the graphic that Martin sent.
> > > > > Since I believe we are not huge enough (yet), lets rule out another
> > > issue
> > > > > by analysing the sql logs (I requested those to Martin offline and
> he
> > > is
> > > > > going to kindly collect them).
> > > > > Also Im curious to know if Mongo behave in the same way.
> > > > >
> > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez Martinez <
> > > > > egonzalez@apache.org> wrote:
> > > > >
> > > > > > Hi Francisco,
> > > > > > I would highly recommend to check indexes and how the updates
> work
> > in
> > > > > data
> > > > > > index to avoid full scan table and lock the full table. Some db
> are
> > > > very
> > > > > > sensitive to that.
> > > > > >
> > > > > > El mié, 7 feb 2024, 18:41, Francisco Javier Tirado Sarti <
> > > > > > ftirados@redhat.com> escribió:
> > > > > >
> > > > > > > Hi Martin,
> > > > > > > While I analyze the data, let me ask you if it is possible to
> > > perform
> > > > > > > another check (similar in a way to disabling data-index like
> you
> > > do)
> > > > > Can
> > > > > > > you switch to MongoDB persistence and check if the same
> > degradation
> > > > > that
> > > > > > is
> > > > > > > there for postgres remains?
> > > > > > > I do not know if this is feasible but will certainly indicate
> the
> > > > > problem
> > > > > > > is on the postgres storage layer and I do not have a clear
> > > prediction
> > > > > of
> > > > > > > what we will see when doing this switch.
> > > > > > >
> > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler
> > > <mweiler@ibm.com.invalid
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Francisco,
> > > > > > > >
> > > > > > > > thanks for your work on this important topic!
> > > > > > > >
> > > > > > > > I would like to share some test results here, which might
> help
> > to
> > > > > > improve
> > > > > > > > the codebase even further. I am using the jmeter based test
> > case
> > > > from
> > > > > > > Pere
> > > > > > > > and Enrique (thanks guys!) [1] which uses a load of 30
> threads
> > to
> > > > > > > >
> > > > > > > > 1) start a new process instance (POST)
> > > > > > > > 2) retrieve tasks for a user (GET)
> > > > > > > > 3) fetches task details (GET)
> > > > > > > > 4) complete a task (POST)
> > > > > > > > 5) execute a query on data-audit
> > > > > > > >
> > > > > > > > With this test setup, I noticed that the performance for the
> > POST
> > > > > > > > requests, in particular the one to start a new process
> > instance,
> > > > > > degrades
> > > > > > > > over time - see graph [2]. If I run the same test without
> > > > data-index,
> > > > > > > then
> > > > > > > > there is no such performance degradation [3]. You can find a
> > > thread
> > > > > > dump
> > > > > > > > captured a few minutes into the first test here [4] that
> might
> > > help
> > > > > to
> > > > > > > see
> > > > > > > > some of the contention points.
> > > > > > > >
> > > > > > > > I'd appreciate if you could take a look and see if there is
> > > > something
> > > > > > > that
> > > > > > > > can be further improved based on your previous work. If you
> > need
> > > > any
> > > > > > > > additional data, let me know, but otherwise it is
> > straightforward
> > > > to
> > > > > > run
> > > > > > > > the jmeter test as well.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Martin
> > > > > > > >
> > > > > > > > [1] https://github.com/pefernan/job-service-refactor-test/
> > > > > > > > [2]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > > > > [3]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > > > > [4]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > > > >
> > > > > > > > ________________________________________
> > > > > > > > From: Francisco Javier Tirado Sarti <ft...@redhat.com>
> > > > > > > > Sent: Wednesday, January 17, 2024 9:13 AM
> > > > > > > > To: dev@kie.apache.org
> > > > > > > > Cc: Pere Fernandez Perez
> > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > > > > data-index
> > > > > > > > persistence addon
> > > > > > > >
> > > > > > > > Hi Alex,
> > > > > > > > I did not take times (which depends on a number of variables
> > that
> > > > > > > > drastically change between environments), but verify that the
> > > > number
> > > > > of
> > > > > > > > updates has been reduced drastically without losing
> > > functionality,
> > > > > > which
> > > > > > > is
> > > > > > > > objectively a good thing. If before the change, for every
> node
> > > > > > executed,
> > > > > > > we
> > > > > > > > have an update for every node previously executed, so if a
> > > process
> > > > > have
> > > > > > > 50
> > > > > > > > nodes to execute, we were performing nearly 50*51/2 updates,
> > > which
> > > > > > gives
> > > > > > > us
> > > > > > > > a total of  1275 updates, now we have just one for every node
> > > being
> > > > > > > > executed, implying a total of 50 updates.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli <
> > alex@porcelli.me>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Francisco,
> > > > > > > > >
> > > > > > > > > I noticed that your PR has been merged, but I was expecting
> > (at
> > > > > least
> > > > > > > > > was my understanding from this thread) that before merging
> > some
> > > > > > > > > benchmark data would be shared in advance - to assess the
> > > > > > cost/benefit
> > > > > > > > > of such a decent size change.
> > > > > > > > >
> > > > > > > > > Do you have any information to share?
> > > > > > > > >
> > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM Francisco Javier Tirado
> Sarti
> > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Yes, as intended, now we have one select and one
> > > insert/update
> > > > > per
> > > > > > > node
> > > > > > > > > > event.
> > > > > > > > > > I moved the PR as ready for review and give @Pere
> Fernandez
> > > > Perez
> > > > > > > > > > <pe...@redhat.com> permission to the branch so he can
> > > edit
> > > > it
> > > > > > in
> > > > > > > > the
> > > > > > > > > > next two weeks (Ill be on PTO)  if desired, before
> merging.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex Porcelli <
> > > > alex@porcelli.me>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Cool, thank you Francisco!
> > > > > > > > > > >
> > > > > > > > > > > Did you manage to get some preliminary data about
> > > > improvements?
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM Francisco Javier
> Tirado
> > > > Sarti
> > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, after some delay because of quarkus 3 migration.
> > Im
> > > > > > refining
> > > > > > > > > this
> > > > > > > > > > > > draft PR
> > > > > > > > > > > >
> > > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex Porcelli <
> > > > > > alex@porcelli.me>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Any update or new findings on this topic?
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM Francisco Javier
> > Tirado
> > > > > Sarti
> > > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > > > > After considering different options to improve
> > > > > performance,
> > > > > > > we
> > > > > > > > > feel
> > > > > > > > > > > that
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > is time to "partially" move away from the current
> > Map
> > > > > style
> > > > > > > > > > > interface (
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > > > > > > > > )
> > > > > > > > > > > > > > which was shared with Trusty, to one more
> suitable
> > > for
> > > > > > usage
> > > > > > > > > with a
> > > > > > > > > > > > > > relational DB like postgresql (but still
> compatible
> > > > with
> > > > > > big
> > > > > > > > > table
> > > > > > > > > > > dbs).
> > > > > > > > > > > > > > The idea will be to replace generic Storage
> > interface
> > > > by
> > > > > > four
> > > > > > > > > > > specific
> > > > > > > > > > > > > > interfaces (which will inherit from a common one
> > that
> > > > > keeps
> > > > > > > the
> > > > > > > > > query
> > > > > > > > > > > > > part
> > > > > > > > > > > > > > at is it. with get and query methods), that will
> > > > include
> > > > > > the
> > > > > > > > > required
> > > > > > > > > > > > > > modification operations for the four DataIndex
> > > > "domains":
> > > > > > > > > > > > > processinstance,
> > > > > > > > > > > > > > usertask, processdefinitions and jobs. Those
> > > interfaces
> > > > > > will
> > > > > > > > > define
> > > > > > > > > > > > > methods
> > > > > > > > > > > > > > like addNode, addVariable, updateTask,
> > > > addAttachment.....
> > > > > > > that
> > > > > > > > > will
> > > > > > > > > > > allow
> > > > > > > > > > > > > > the persistent layer implementation  to just
> update
> > > the
> > > > > > > needed
> > > > > > > > > info
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > DB  (for example, for addNode in Postgres, just
> > > insert
> > > > a
> > > > > > row
> > > > > > > > into
> > > > > > > > > > > nodes
> > > > > > > > > > > > > > table, for addNode in Mongo, basically the same
> > > atomic
> > > > > > upsert
> > > > > > > > > > > operation
> > > > > > > > > > > > > > that is currently done). Therefore, we increase
> > > > > performance
> > > > > > > for
> > > > > > > > > > > Postgres
> > > > > > > > > > > > > > and keep the current one for Mongo. The current
> DB
> > > > > schemas
> > > > > > > > won't
> > > > > > > > > be
> > > > > > > > > > > > > > touched.
> > > > > > > > > > > > > > Since the code change is large, I do not think
> I'll
> > > be
> > > > > able
> > > > > > > to
> > > > > > > > > have
> > > > > > > > > > > the
> > > > > > > > > > > > > PR
> > > > > > > > > > > > > > ready till next week.
> > > > > > > > > > > > > > But before starting, please let me know if that
> > > > approach
> > > > > is
> > > > > > > > fine
> > > > > > > > > for
> > > > > > > > > > > you.
> > > > > > > > > > > > > > Best regards.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 6:55 PM Alex Porcelli <
> > > > > > > > alex@porcelli.me>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thank you Francisco to getting deeper on this…
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Looking forward to see the results of your
> > > suggested
> > > > > > > > > improvements.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 9:40 AM Francisco
> Javier
> > > > Tirado
> > > > > > > > Sarti <
> > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I forgot to attach the queries
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 3:04 PM Francisco
> > Javier
> > > > > Tirado
> > > > > > > > > Sarti <
> > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >> Hi,
> > > > > > > > > > > > > > > >> A brief update on this topic.
> > > > > > > > > > > > > > > >> After doing a simple test with example
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > > > > > > > > > ,
> > > > > > > > > > > > > > > >> the number of updates over Nodes table is
> n*n,
> > > so
> > > > we
> > > > > > > > manage
> > > > > > > > > to
> > > > > > > > > > > > > obtain a
> > > > > > > > > > > > > > > >> perfect quadratic performance degradation.
> The
> > > > > problem
> > > > > > > is
> > > > > > > > > worse
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > >> of Serverless Workflow than in BPMN because
> we
> > > the
> > > > > > > number
> > > > > > > > of
> > > > > > > > > > > nodes
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > >> greater than the number of states. In that
> > > > example N
> > > > > > is
> > > > > > > > 16,
> > > > > > > > > but
> > > > > > > > > > > for
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > >> complex workflow it would be certainly
> large.
> > > > > > > > > > > > > > > >> I think that this is more related to how we
> > are
> > > > > > handling
> > > > > > > > > JPA in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > code,
> > > > > > > > > > > > > > > >> in particular the mapping from model to
> entity
> > > > > > > (basically
> > > > > > > > > JPA is
> > > > > > > > > > > > > blind
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > >> has to update all nodes for every write
> > because
> > > it
> > > > > > > > believes
> > > > > > > > > the
> > > > > > > > > > > > > node has
> > > > > > > > > > > > > > > >> been updated, although it is not) than an
> > issue
> > > in
> > > > > the
> > > > > > > > table
> > > > > > > > > > > > > definition.
> > > > > > > > > > > > > > > >> In fact, when using JPA, separating the
> server
> > > > model
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > > > JPA
> > > > > > > > > > > > > > > entity is
> > > > > > > > > > > > > > > >> not a good idea, especially if the entity
> > > contains
> > > > > > > > > collections.
> > > > > > > > > > > I
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > try
> > > > > > > > > > > > > > > >> to change that without breaking anything.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> On Wed, Nov 22, 2023 at 12:10 PM Enrique
> > > Gonzalez
> > > > > > > > Martinez <
> > > > > > > > > > > > > > > >> egonzalez@apache.org> wrote:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >>> After the events split you now will need to
> > > > create
> > > > > a
> > > > > > > node
> > > > > > > > > > > instance
> > > > > > > > > > > > > > > >>> model instance of making independent from
> the
> > > > > process
> > > > > > > > > instance.
> > > > > > > > > > > > > > > >>> That should do the trick.
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>> Regarding deleting/inserting it was fixed
> at
> > > some
> > > > > > > point.
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>> El mar, 21 nov 2023 a las 20:22, Francisco
> > > Javier
> > > > > > > Tirado
> > > > > > > > > Sarti
> > > > > > > > > > > > > > > >>> (<ft...@redhat.com>) escribió:
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > Hi Martin,
> > > > > > > > > > > > > > > >>> > I have a task to review performance of
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > ProcessInstanceNodeDataEventMerger
> > > > > > > > > > > > > > > >>> > My idea is to reduce the number of delete
> > > > inserts
> > > > > > > when
> > > > > > > > > > > processing
> > > > > > > > > > > > > > > >>> events
> > > > > > > > > > > > > > > >>> > and try to do it incremental.
> > > > > > > > > > > > > > > >>> > That should improve performance.
> > > > > > > > > > > > > > > >>> > PS:
> > > > > > > > > > > > > > > >>> > I was planning to send an e-mail tomorrow
> > > > > > announcing
> > > > > > > > > that in
> > > > > > > > > > > > > case you
> > > > > > > > > > > > > > > >>> were
> > > > > > > > > > > > > > > >>> > already working on a fix for that. I
> assume
> > > you
> > > > > are
> > > > > > > not
> > > > > > > > > and I
> > > > > > > > > > > > > would
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > >>> > sending a PR soon.
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > On Tue, Nov 21, 2023 at 6:09 PM Martin
> > Weiler
> > > > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > wrote:
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > > I looked into the new examples using
> > > > data-index
> > > > > > > > > persistence
> > > > > > > > > > > > > addon -
> > > > > > > > > > > > > > > >>> Neus'
> > > > > > > > > > > > > > > >>> > > PR#1813 [1] for serverless and Pere's
> > > branch
> > > > > [2]
> > > > > > > for
> > > > > > > > > > > workflow
> > > > > > > > > > > > > > > (great
> > > > > > > > > > > > > > > >>> job
> > > > > > > > > > > > > > > >>> > > both!) - and they work without issues
> > using
> > > > > > single
> > > > > > > > > > > requests.
> > > > > > > > > > > > > > > >>> However, under
> > > > > > > > > > > > > > > >>> > > some load (I used 'ab' for testing
> with a
> > > > light
> > > > > > > > > > > concurrency of
> > > > > > > > > > > > > 10
> > > > > > > > > > > > > > > >>> parallel
> > > > > > > > > > > > > > > >>> > > requests) I ran into the following
> > > problems:
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > > (1) Large number of insert/delete calls
> > > (eg.
> > > > > for
> > > > > > > > tables
> > > > > > > > > > > such as
> > > > > > > > > > > > > > > >>> nodes,
> > > > > > > > > > > > > > > >>> > > definitions, etc)
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > > (2) Hibernate OptimisticLockExceptions
> /
> > > > > > > > > > > StaleStateExceptions
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > > (3) DB deadlocks
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > > (4) Error responses, slow response
> times
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > > The reason I am reaching out with this
> > > topic
> > > > > here
> > > > > > > is
> > > > > > > > to
> > > > > > > > > > > find
> > > > > > > > > > > > > out if
> > > > > > > > > > > > > > > >>> we are
> > > > > > > > > > > > > > > >>> > > aware of this issue, and if someone is
> > > > already
> > > > > > > > looking
> > > > > > > > > > > into or
> > > > > > > > > > > > > > > being
> > > > > > > > > > > > > > > >>> > > assigned to it?
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > > Thanks,
> > > > > > > > > > > > > > > >>> > > Martin
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > > [1]
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > >
> > > > > >
> https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > > > > > > > > > > > >>> > > [2]
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > > > > > > > > >>> > > To unsubscribe, e-mail:
> > > > > > > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > > > > >>> > > For additional commands, e-mail:
> > > > > > > > > dev-help@kie.apache.org
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > > > > > > > >>> To unsubscribe, e-mail:
> > > > > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > > > > >>> For additional commands, e-mail:
> > > > > > > dev-help@kie.apache.org
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > > > > > For additional commands, e-mail:
> > > > > > dev-help@kie.apache.org
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > > > > > To unsubscribe, e-mail:
> > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > > For additional commands, e-mail:
> > > dev-help@kie.apache.org
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > > > > For additional commands, e-mail:
> dev-help@kie.apache.org
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

RE: [DISCUSSION] Performance issues with data-index persistence addon

[Martin Weiler <mw...@ibm.com.INVALID>]

Thanks everyone for your input. Based on this discussion, I opened the following PR:
https://github.com/apache/incubator-kie-kogito-apps/pull/1985

With this change, the performance seems to be stable over time:
https://drive.google.com/file/d/1zkullvfrJpRp7TRjxDa41ok6kEIR7Fty/view?usp=sharing

Martin

________________________________________
From: Gonzalo Muñoz <gm...@apache.org>
Sent: Friday, February 9, 2024 9:42 AM
To: dev@kie.apache.org
Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with data-index persistence addon

Great work Francisco,
Martin, take a look at this link with some related tips (in case you find
it useful): https://www.cybertec-postgresql.com/en/index-your-foreign-key/

El vie, 9 feb 2024 a las 17:20, Francisco Javier Tirado Sarti (<
ftirados@redhat.com>) escribió:

> For the moment being, we will keep JPA till we exhaust all possibilities,
> let's call switching from jpa to jdbc our hidden plan B ;)
> I already told Martin, but in order everyone to know, just after writing
> the previous email, I thought "what if Postgres is not automatically
> indexing foreign keys like mysql?" and, eureka
> Postgres doc
> https://www.postgresql.org/docs/current/ddl-constraints.html
> Mysql doc
> https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> These are the relevant excerpt
>
> *Postgresql*
> *A foreign key must reference columns that either are a primary key or form
> a unique constraint, or are columns from a non-partial unique index. This
> means that the referenced columns always have an index to allow efficient
> lookups on whether a referencing row has a match. Since a DELETE of a row
> from the referenced table or an UPDATE of a referenced column will require
> a scan of the referencing table for rows matching the old value, it is
> often a good idea to index the referencing columns too. Because this is not
> always needed, and there are many choices available on how to index, the
> declaration of a foreign key constraint does not automatically create an
> index on the referencing columns.*
> *Mysql*
> *MySQL requires that foreign key columns be indexed; if you create a table
> with a foreign key constraint but no index on a given column, an index is
> created. *
>
> So I asked Martin to especially create an index for process_instance_id
> column on nodes table
> I think that will fix the problem detected on the thread dump.
> The simpler process test to verify queries are fine still stands, though ;)
>
>
> On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <tz...@apache.org> wrote:
>
> > I always preferred pure JDBC over Hibernate myself, just for the sake of
> > control of what is happening :) So I would not -1 that myself.
> >
> > Tibor
> >
> > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado Sarti <
> > ftirados@redhat.com>
> > napísal(a):
> >
> > > Hi,
> > > Usually I do not want to talk about work in progress because
> preliminary
> > > conclusions are pretty volatile but, well, there are a couple of things
> > > that can be concluded from the really valuable information that Martin
> > > provided.
> > > 1) In order to be able to determine if the number of statements is
> larger
> > > than expected, I asked Martin to test with a simpler process
> definition.
> > > One with just three nodes: start, script and end. The script one should
> > > change just one variable. This way we can analyze if the number of
> > queries
> > > is the expected one. From the single log (audit was activated them) my
> > > conclusion is that the number of insert/updates over processes and
> nodes
> > > (there a lot over task, that I will prefer to skip for now, baby steps)
> > is
> > > the expected one.
> > > 2) Analysing the thread dump, we see around 15 threads executing this
> > line
> > > at
> > >
> > >
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > so its pretty clear the code to be optimized ;). I'm evaluating
> > > possibilities within JPA/Hibernate, but I'm starting to think that it
> > might
> > > be better to switch to JDBC and skip hibernate. Our lives will be
> > simpler,
> > > especially with a schema relatively simple like ours (that will be my
> > > recommendation if I was an external consultant)
> > >
> > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <tz...@apache.org>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > this will be a bit off-topic. However as far as performance, I think
> we
> > > > should think about that we have string primary keys (IDs). I would
> > expect
> > > > the database systems are much better with indexing numeric keys than
> > > > strings. I remember from the past, when I was working with DBs, that
> > > using
> > > > strings as keys or indexes was a discouraged practice.
> > > >
> > > > Best regards,
> > > > Tibor
> > > >
> > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler <mw...@ibm.com.invalid>
> > > > napísal(a):
> > > >
> > > > > I changed the test to use MongoDB [1] and I don't see a performance
> > > > > degradation with this setup [2].
> > > > >
> > > > > Please keep us posted of your findings. Thanks!
> > > > >
> > > > > Martin
> > > > >
> > > > > [1]
> > > >
> https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > > [2]
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > >
> > > > > ________________________________________
> > > > > From: Francisco Javier Tirado Sarti <ft...@redhat.com>
> > > > > Sent: Wednesday, February 7, 2024 11:40 AM
> > > > > To: dev@kie.apache.org
> > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > data-index
> > > > > persistence addon
> > > > >
> > > > > yes, it can be index degradation because of size, but I believe (I
> > > might
> > > > be
> > > > > wrong) the db is too small (yet) for that.
> > > > > But, eventually, Postgres, when the DB is huge enough, unavoidably
> > will
> > > > > behave like the graphic that Martin sent.
> > > > > Since I believe we are not huge enough (yet), lets rule out another
> > > issue
> > > > > by analysing the sql logs (I requested those to Martin offline and
> he
> > > is
> > > > > going to kindly collect them).
> > > > > Also Im curious to know if Mongo behave in the same way.
> > > > >
> > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez Martinez <
> > > > > egonzalez@apache.org> wrote:
> > > > >
> > > > > > Hi Francisco,
> > > > > > I would highly recommend to check indexes and how the updates
> work
> > in
> > > > > data
> > > > > > index to avoid full scan table and lock the full table. Some db
> are
> > > > very
> > > > > > sensitive to that.
> > > > > >
> > > > > > El mié, 7 feb 2024, 18:41, Francisco Javier Tirado Sarti <
> > > > > > ftirados@redhat.com> escribió:
> > > > > >
> > > > > > > Hi Martin,
> > > > > > > While I analyze the data, let me ask you if it is possible to
> > > perform
> > > > > > > another check (similar in a way to disabling data-index like
> you
> > > do)
> > > > > Can
> > > > > > > you switch to MongoDB persistence and check if the same
> > degradation
> > > > > that
> > > > > > is
> > > > > > > there for postgres remains?
> > > > > > > I do not know if this is feasible but will certainly indicate
> the
> > > > > problem
> > > > > > > is on the postgres storage layer and I do not have a clear
> > > prediction
> > > > > of
> > > > > > > what we will see when doing this switch.
> > > > > > >
> > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler
> > > <mweiler@ibm.com.invalid
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Francisco,
> > > > > > > >
> > > > > > > > thanks for your work on this important topic!
> > > > > > > >
> > > > > > > > I would like to share some test results here, which might
> help
> > to
> > > > > > improve
> > > > > > > > the codebase even further. I am using the jmeter based test
> > case
> > > > from
> > > > > > > Pere
> > > > > > > > and Enrique (thanks guys!) [1] which uses a load of 30
> threads
> > to
> > > > > > > >
> > > > > > > > 1) start a new process instance (POST)
> > > > > > > > 2) retrieve tasks for a user (GET)
> > > > > > > > 3) fetches task details (GET)
> > > > > > > > 4) complete a task (POST)
> > > > > > > > 5) execute a query on data-audit
> > > > > > > >
> > > > > > > > With this test setup, I noticed that the performance for the
> > POST
> > > > > > > > requests, in particular the one to start a new process
> > instance,
> > > > > > degrades
> > > > > > > > over time - see graph [2]. If I run the same test without
> > > > data-index,
> > > > > > > then
> > > > > > > > there is no such performance degradation [3]. You can find a
> > > thread
> > > > > > dump
> > > > > > > > captured a few minutes into the first test here [4] that
> might
> > > help
> > > > > to
> > > > > > > see
> > > > > > > > some of the contention points.
> > > > > > > >
> > > > > > > > I'd appreciate if you could take a look and see if there is
> > > > something
> > > > > > > that
> > > > > > > > can be further improved based on your previous work. If you
> > need
> > > > any
> > > > > > > > additional data, let me know, but otherwise it is
> > straightforward
> > > > to
> > > > > > run
> > > > > > > > the jmeter test as well.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Martin
> > > > > > > >
> > > > > > > > [1] https://github.com/pefernan/job-service-refactor-test/
> > > > > > > > [2]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > > > > [3]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > > > > [4]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > > > >
> > > > > > > > ________________________________________
> > > > > > > > From: Francisco Javier Tirado Sarti <ft...@redhat.com>
> > > > > > > > Sent: Wednesday, January 17, 2024 9:13 AM
> > > > > > > > To: dev@kie.apache.org
> > > > > > > > Cc: Pere Fernandez Perez
> > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > > > > data-index
> > > > > > > > persistence addon
> > > > > > > >
> > > > > > > > Hi Alex,
> > > > > > > > I did not take times (which depends on a number of variables
> > that
> > > > > > > > drastically change between environments), but verify that the
> > > > number
> > > > > of
> > > > > > > > updates has been reduced drastically without losing
> > > functionality,
> > > > > > which
> > > > > > > is
> > > > > > > > objectively a good thing. If before the change, for every
> node
> > > > > > executed,
> > > > > > > we
> > > > > > > > have an update for every node previously executed, so if a
> > > process
> > > > > have
> > > > > > > 50
> > > > > > > > nodes to execute, we were performing nearly 50*51/2 updates,
> > > which
> > > > > > gives
> > > > > > > us
> > > > > > > > a total of  1275 updates, now we have just one for every node
> > > being
> > > > > > > > executed, implying a total of 50 updates.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli <
> > alex@porcelli.me>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Francisco,
> > > > > > > > >
> > > > > > > > > I noticed that your PR has been merged, but I was expecting
> > (at
> > > > > least
> > > > > > > > > was my understanding from this thread) that before merging
> > some
> > > > > > > > > benchmark data would be shared in advance - to assess the
> > > > > > cost/benefit
> > > > > > > > > of such a decent size change.
> > > > > > > > >
> > > > > > > > > Do you have any information to share?
> > > > > > > > >
> > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM Francisco Javier Tirado
> Sarti
> > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Yes, as intended, now we have one select and one
> > > insert/update
> > > > > per
> > > > > > > node
> > > > > > > > > > event.
> > > > > > > > > > I moved the PR as ready for review and give @Pere
> Fernandez
> > > > Perez
> > > > > > > > > > <pe...@redhat.com> permission to the branch so he can
> > > edit
> > > > it
> > > > > > in
> > > > > > > > the
> > > > > > > > > > next two weeks (Ill be on PTO)  if desired, before
> merging.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex Porcelli <
> > > > alex@porcelli.me>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Cool, thank you Francisco!
> > > > > > > > > > >
> > > > > > > > > > > Did you manage to get some preliminary data about
> > > > improvements?
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM Francisco Javier
> Tirado
> > > > Sarti
> > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, after some delay because of quarkus 3 migration.
> > Im
> > > > > > refining
> > > > > > > > > this
> > > > > > > > > > > > draft PR
> > > > > > > > > > > >
> > > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex Porcelli <
> > > > > > alex@porcelli.me>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Any update or new findings on this topic?
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM Francisco Javier
> > Tirado
> > > > > Sarti
> > > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > > > > After considering different options to improve
> > > > > performance,
> > > > > > > we
> > > > > > > > > feel
> > > > > > > > > > > that
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > is time to "partially" move away from the current
> > Map
> > > > > style
> > > > > > > > > > > interface (
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > > > > > > > > )
> > > > > > > > > > > > > > which was shared with Trusty, to one more
> suitable
> > > for
> > > > > > usage
> > > > > > > > > with a
> > > > > > > > > > > > > > relational DB like postgresql (but still
> compatible
> > > > with
> > > > > > big
> > > > > > > > > table
> > > > > > > > > > > dbs).
> > > > > > > > > > > > > > The idea will be to replace generic Storage
> > interface
> > > > by
> > > > > > four
> > > > > > > > > > > specific
> > > > > > > > > > > > > > interfaces (which will inherit from a common one
> > that
> > > > > keeps
> > > > > > > the
> > > > > > > > > query
> > > > > > > > > > > > > part
> > > > > > > > > > > > > > at is it. with get and query methods), that will
> > > > include
> > > > > > the
> > > > > > > > > required
> > > > > > > > > > > > > > modification operations for the four DataIndex
> > > > "domains":
> > > > > > > > > > > > > processinstance,
> > > > > > > > > > > > > > usertask, processdefinitions and jobs. Those
> > > interfaces
> > > > > > will
> > > > > > > > > define
> > > > > > > > > > > > > methods
> > > > > > > > > > > > > > like addNode, addVariable, updateTask,
> > > > addAttachment.....
> > > > > > > that
> > > > > > > > > will
> > > > > > > > > > > allow
> > > > > > > > > > > > > > the persistent layer implementation  to just
> update
> > > the
> > > > > > > needed
> > > > > > > > > info
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > DB  (for example, for addNode in Postgres, just
> > > insert
> > > > a
> > > > > > row
> > > > > > > > into
> > > > > > > > > > > nodes
> > > > > > > > > > > > > > table, for addNode in Mongo, basically the same
> > > atomic
> > > > > > upsert
> > > > > > > > > > > operation
> > > > > > > > > > > > > > that is currently done). Therefore, we increase
> > > > > performance
> > > > > > > for
> > > > > > > > > > > Postgres
> > > > > > > > > > > > > > and keep the current one for Mongo. The current
> DB
> > > > > schemas
> > > > > > > > won't
> > > > > > > > > be
> > > > > > > > > > > > > > touched.
> > > > > > > > > > > > > > Since the code change is large, I do not think
> I'll
> > > be
> > > > > able
> > > > > > > to
> > > > > > > > > have
> > > > > > > > > > > the
> > > > > > > > > > > > > PR
> > > > > > > > > > > > > > ready till next week.
> > > > > > > > > > > > > > But before starting, please let me know if that
> > > > approach
> > > > > is
> > > > > > > > fine
> > > > > > > > > for
> > > > > > > > > > > you.
> > > > > > > > > > > > > > Best regards.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 6:55 PM Alex Porcelli <
> > > > > > > > alex@porcelli.me>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thank you Francisco to getting deeper on this…
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Looking forward to see the results of your
> > > suggested
> > > > > > > > > improvements.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 9:40 AM Francisco
> Javier
> > > > Tirado
> > > > > > > > Sarti <
> > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I forgot to attach the queries
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 3:04 PM Francisco
> > Javier
> > > > > Tirado
> > > > > > > > > Sarti <
> > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >> Hi,
> > > > > > > > > > > > > > > >> A brief update on this topic.
> > > > > > > > > > > > > > > >> After doing a simple test with example
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > > > > > > > > > ,
> > > > > > > > > > > > > > > >> the number of updates over Nodes table is
> n*n,
> > > so
> > > > we
> > > > > > > > manage
> > > > > > > > > to
> > > > > > > > > > > > > obtain a
> > > > > > > > > > > > > > > >> perfect quadratic performance degradation.
> The
> > > > > problem
> > > > > > > is
> > > > > > > > > worse
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > >> of Serverless Workflow than in BPMN because
> we
> > > the
> > > > > > > number
> > > > > > > > of
> > > > > > > > > > > nodes
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > >> greater than the number of states. In that
> > > > example N
> > > > > > is
> > > > > > > > 16,
> > > > > > > > > but
> > > > > > > > > > > for
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > >> complex workflow it would be certainly
> large.
> > > > > > > > > > > > > > > >> I think that this is more related to how we
> > are
> > > > > > handling
> > > > > > > > > JPA in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > code,
> > > > > > > > > > > > > > > >> in particular the mapping from model to
> entity
> > > > > > > (basically
> > > > > > > > > JPA is
> > > > > > > > > > > > > blind
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > >> has to update all nodes for every write
> > because
> > > it
> > > > > > > > believes
> > > > > > > > > the
> > > > > > > > > > > > > node has
> > > > > > > > > > > > > > > >> been updated, although it is not) than an
> > issue
> > > in
> > > > > the
> > > > > > > > table
> > > > > > > > > > > > > definition.
> > > > > > > > > > > > > > > >> In fact, when using JPA, separating the
> server
> > > > model
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > > > JPA
> > > > > > > > > > > > > > > entity is
> > > > > > > > > > > > > > > >> not a good idea, especially if the entity
> > > contains
> > > > > > > > > collections.
> > > > > > > > > > > I
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > try
> > > > > > > > > > > > > > > >> to change that without breaking anything.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> On Wed, Nov 22, 2023 at 12:10 PM Enrique
> > > Gonzalez
> > > > > > > > Martinez <
> > > > > > > > > > > > > > > >> egonzalez@apache.org> wrote:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >>> After the events split you now will need to
> > > > create
> > > > > a
> > > > > > > node
> > > > > > > > > > > instance
> > > > > > > > > > > > > > > >>> model instance of making independent from
> the
> > > > > process
> > > > > > > > > instance.
> > > > > > > > > > > > > > > >>> That should do the trick.
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>> Regarding deleting/inserting it was fixed
> at
> > > some
> > > > > > > point.
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>> El mar, 21 nov 2023 a las 20:22, Francisco
> > > Javier
> > > > > > > Tirado
> > > > > > > > > Sarti
> > > > > > > > > > > > > > > >>> (<ft...@redhat.com>) escribió:
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > Hi Martin,
> > > > > > > > > > > > > > > >>> > I have a task to review performance of
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > ProcessInstanceNodeDataEventMerger
> > > > > > > > > > > > > > > >>> > My idea is to reduce the number of delete
> > > > inserts
> > > > > > > when
> > > > > > > > > > > processing
> > > > > > > > > > > > > > > >>> events
> > > > > > > > > > > > > > > >>> > and try to do it incremental.
> > > > > > > > > > > > > > > >>> > That should improve performance.
> > > > > > > > > > > > > > > >>> > PS:
> > > > > > > > > > > > > > > >>> > I was planning to send an e-mail tomorrow
> > > > > > announcing
> > > > > > > > > that in
> > > > > > > > > > > > > case you
> > > > > > > > > > > > > > > >>> were
> > > > > > > > > > > > > > > >>> > already working on a fix for that. I
> assume
> > > you
> > > > > are
> > > > > > > not
> > > > > > > > > and I
> > > > > > > > > > > > > would
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > >>> > sending a PR soon.
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > On Tue, Nov 21, 2023 at 6:09 PM Martin
> > Weiler
> > > > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > wrote:
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > > I looked into the new examples using
> > > > data-index
> > > > > > > > > persistence
> > > > > > > > > > > > > addon -
> > > > > > > > > > > > > > > >>> Neus'
> > > > > > > > > > > > > > > >>> > > PR#1813 [1] for serverless and Pere's
> > > branch
> > > > > [2]
> > > > > > > for
> > > > > > > > > > > workflow
> > > > > > > > > > > > > > > (great
> > > > > > > > > > > > > > > >>> job
> > > > > > > > > > > > > > > >>> > > both!) - and they work without issues
> > using
> > > > > > single
> > > > > > > > > > > requests.
> > > > > > > > > > > > > > > >>> However, under
> > > > > > > > > > > > > > > >>> > > some load (I used 'ab' for testing
> with a
> > > > light
> > > > > > > > > > > concurrency of
> > > > > > > > > > > > > 10
> > > > > > > > > > > > > > > >>> parallel
> > > > > > > > > > > > > > > >>> > > requests) I ran into the following
> > > problems:
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > > (1) Large number of insert/delete calls
> > > (eg.
> > > > > for
> > > > > > > > tables
> > > > > > > > > > > such as
> > > > > > > > > > > > > > > >>> nodes,
> > > > > > > > > > > > > > > >>> > > definitions, etc)
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > > (2) Hibernate OptimisticLockExceptions
> /
> > > > > > > > > > > StaleStateExceptions
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > > (3) DB deadlocks
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > > (4) Error responses, slow response
> times
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > > The reason I am reaching out with this
> > > topic
> > > > > here
> > > > > > > is
> > > > > > > > to
> > > > > > > > > > > find
> > > > > > > > > > > > > out if
> > > > > > > > > > > > > > > >>> we are
> > > > > > > > > > > > > > > >>> > > aware of this issue, and if someone is
> > > > already
> > > > > > > > looking
> > > > > > > > > > > into or
> > > > > > > > > > > > > > > being
> > > > > > > > > > > > > > > >>> > > assigned to it?
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > > Thanks,
> > > > > > > > > > > > > > > >>> > > Martin
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > > [1]
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > >
> > > > > >
> https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > > > > > > > > > > > >>> > > [2]
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > > > > > > > > >>> > > To unsubscribe, e-mail:
> > > > > > > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > > > > >>> > > For additional commands, e-mail:
> > > > > > > > > dev-help@kie.apache.org
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > > > > > > > >>> To unsubscribe, e-mail:
> > > > > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > > > > >>> For additional commands, e-mail:
> > > > > > > dev-help@kie.apache.org
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > > > > > For additional commands, e-mail:
> > > > > > dev-help@kie.apache.org
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > > > > > To unsubscribe, e-mail:
> > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > > For additional commands, e-mail:
> > > dev-help@kie.apache.org
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > > > > For additional commands, e-mail:
> dev-help@kie.apache.org
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Gonzalo Muñoz <gm...@apache.org>]

Great work Francisco,
Martin, take a look at this link with some related tips (in case you find
it useful): https://www.cybertec-postgresql.com/en/index-your-foreign-key/

El vie, 9 feb 2024 a las 17:20, Francisco Javier Tirado Sarti (<
ftirados@redhat.com>) escribió:

> For the moment being, we will keep JPA till we exhaust all possibilities,
> let's call switching from jpa to jdbc our hidden plan B ;)
> I already told Martin, but in order everyone to know, just after writing
> the previous email, I thought "what if Postgres is not automatically
> indexing foreign keys like mysql?" and, eureka
> Postgres doc
> https://www.postgresql.org/docs/current/ddl-constraints.html
> Mysql doc
> https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
> These are the relevant excerpt
>
> *Postgresql*
> *A foreign key must reference columns that either are a primary key or form
> a unique constraint, or are columns from a non-partial unique index. This
> means that the referenced columns always have an index to allow efficient
> lookups on whether a referencing row has a match. Since a DELETE of a row
> from the referenced table or an UPDATE of a referenced column will require
> a scan of the referencing table for rows matching the old value, it is
> often a good idea to index the referencing columns too. Because this is not
> always needed, and there are many choices available on how to index, the
> declaration of a foreign key constraint does not automatically create an
> index on the referencing columns.*
> *Mysql*
> *MySQL requires that foreign key columns be indexed; if you create a table
> with a foreign key constraint but no index on a given column, an index is
> created. *
>
> So I asked Martin to especially create an index for process_instance_id
> column on nodes table
> I think that will fix the problem detected on the thread dump.
> The simpler process test to verify queries are fine still stands, though ;)
>
>
> On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <tz...@apache.org> wrote:
>
> > I always preferred pure JDBC over Hibernate myself, just for the sake of
> > control of what is happening :) So I would not -1 that myself.
> >
> > Tibor
> >
> > Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado Sarti <
> > ftirados@redhat.com>
> > napísal(a):
> >
> > > Hi,
> > > Usually I do not want to talk about work in progress because
> preliminary
> > > conclusions are pretty volatile but, well, there are a couple of things
> > > that can be concluded from the really valuable information that Martin
> > > provided.
> > > 1) In order to be able to determine if the number of statements is
> larger
> > > than expected, I asked Martin to test with a simpler process
> definition.
> > > One with just three nodes: start, script and end. The script one should
> > > change just one variable. This way we can analyze if the number of
> > queries
> > > is the expected one. From the single log (audit was activated them) my
> > > conclusion is that the number of insert/updates over processes and
> nodes
> > > (there a lot over task, that I will prefer to skip for now, baby steps)
> > is
> > > the expected one.
> > > 2) Analysing the thread dump, we see around 15 threads executing this
> > line
> > > at
> > >
> > >
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > > so its pretty clear the code to be optimized ;). I'm evaluating
> > > possibilities within JPA/Hibernate, but I'm starting to think that it
> > might
> > > be better to switch to JDBC and skip hibernate. Our lives will be
> > simpler,
> > > especially with a schema relatively simple like ours (that will be my
> > > recommendation if I was an external consultant)
> > >
> > > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <tz...@apache.org>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > this will be a bit off-topic. However as far as performance, I think
> we
> > > > should think about that we have string primary keys (IDs). I would
> > expect
> > > > the database systems are much better with indexing numeric keys than
> > > > strings. I remember from the past, when I was working with DBs, that
> > > using
> > > > strings as keys or indexes was a discouraged practice.
> > > >
> > > > Best regards,
> > > > Tibor
> > > >
> > > > Dňa št 8. 2. 2024, 22:45 Martin Weiler <mw...@ibm.com.invalid>
> > > > napísal(a):
> > > >
> > > > > I changed the test to use MongoDB [1] and I don't see a performance
> > > > > degradation with this setup [2].
> > > > >
> > > > > Please keep us posted of your findings. Thanks!
> > > > >
> > > > > Martin
> > > > >
> > > > > [1]
> > > >
> https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > > [2]
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > > >
> > > > > ________________________________________
> > > > > From: Francisco Javier Tirado Sarti <ft...@redhat.com>
> > > > > Sent: Wednesday, February 7, 2024 11:40 AM
> > > > > To: dev@kie.apache.org
> > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > data-index
> > > > > persistence addon
> > > > >
> > > > > yes, it can be index degradation because of size, but I believe (I
> > > might
> > > > be
> > > > > wrong) the db is too small (yet) for that.
> > > > > But, eventually, Postgres, when the DB is huge enough, unavoidably
> > will
> > > > > behave like the graphic that Martin sent.
> > > > > Since I believe we are not huge enough (yet), lets rule out another
> > > issue
> > > > > by analysing the sql logs (I requested those to Martin offline and
> he
> > > is
> > > > > going to kindly collect them).
> > > > > Also Im curious to know if Mongo behave in the same way.
> > > > >
> > > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez Martinez <
> > > > > egonzalez@apache.org> wrote:
> > > > >
> > > > > > Hi Francisco,
> > > > > > I would highly recommend to check indexes and how the updates
> work
> > in
> > > > > data
> > > > > > index to avoid full scan table and lock the full table. Some db
> are
> > > > very
> > > > > > sensitive to that.
> > > > > >
> > > > > > El mié, 7 feb 2024, 18:41, Francisco Javier Tirado Sarti <
> > > > > > ftirados@redhat.com> escribió:
> > > > > >
> > > > > > > Hi Martin,
> > > > > > > While I analyze the data, let me ask you if it is possible to
> > > perform
> > > > > > > another check (similar in a way to disabling data-index like
> you
> > > do)
> > > > > Can
> > > > > > > you switch to MongoDB persistence and check if the same
> > degradation
> > > > > that
> > > > > > is
> > > > > > > there for postgres remains?
> > > > > > > I do not know if this is feasible but will certainly indicate
> the
> > > > > problem
> > > > > > > is on the postgres storage layer and I do not have a clear
> > > prediction
> > > > > of
> > > > > > > what we will see when doing this switch.
> > > > > > >
> > > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler
> > > <mweiler@ibm.com.invalid
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Francisco,
> > > > > > > >
> > > > > > > > thanks for your work on this important topic!
> > > > > > > >
> > > > > > > > I would like to share some test results here, which might
> help
> > to
> > > > > > improve
> > > > > > > > the codebase even further. I am using the jmeter based test
> > case
> > > > from
> > > > > > > Pere
> > > > > > > > and Enrique (thanks guys!) [1] which uses a load of 30
> threads
> > to
> > > > > > > >
> > > > > > > > 1) start a new process instance (POST)
> > > > > > > > 2) retrieve tasks for a user (GET)
> > > > > > > > 3) fetches task details (GET)
> > > > > > > > 4) complete a task (POST)
> > > > > > > > 5) execute a query on data-audit
> > > > > > > >
> > > > > > > > With this test setup, I noticed that the performance for the
> > POST
> > > > > > > > requests, in particular the one to start a new process
> > instance,
> > > > > > degrades
> > > > > > > > over time - see graph [2]. If I run the same test without
> > > > data-index,
> > > > > > > then
> > > > > > > > there is no such performance degradation [3]. You can find a
> > > thread
> > > > > > dump
> > > > > > > > captured a few minutes into the first test here [4] that
> might
> > > help
> > > > > to
> > > > > > > see
> > > > > > > > some of the contention points.
> > > > > > > >
> > > > > > > > I'd appreciate if you could take a look and see if there is
> > > > something
> > > > > > > that
> > > > > > > > can be further improved based on your previous work. If you
> > need
> > > > any
> > > > > > > > additional data, let me know, but otherwise it is
> > straightforward
> > > > to
> > > > > > run
> > > > > > > > the jmeter test as well.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Martin
> > > > > > > >
> > > > > > > > [1] https://github.com/pefernan/job-service-refactor-test/
> > > > > > > > [2]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > > > > [3]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > > > > [4]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > > > >
> > > > > > > > ________________________________________
> > > > > > > > From: Francisco Javier Tirado Sarti <ft...@redhat.com>
> > > > > > > > Sent: Wednesday, January 17, 2024 9:13 AM
> > > > > > > > To: dev@kie.apache.org
> > > > > > > > Cc: Pere Fernandez Perez
> > > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > > > > data-index
> > > > > > > > persistence addon
> > > > > > > >
> > > > > > > > Hi Alex,
> > > > > > > > I did not take times (which depends on a number of variables
> > that
> > > > > > > > drastically change between environments), but verify that the
> > > > number
> > > > > of
> > > > > > > > updates has been reduced drastically without losing
> > > functionality,
> > > > > > which
> > > > > > > is
> > > > > > > > objectively a good thing. If before the change, for every
> node
> > > > > > executed,
> > > > > > > we
> > > > > > > > have an update for every node previously executed, so if a
> > > process
> > > > > have
> > > > > > > 50
> > > > > > > > nodes to execute, we were performing nearly 50*51/2 updates,
> > > which
> > > > > > gives
> > > > > > > us
> > > > > > > > a total of  1275 updates, now we have just one for every node
> > > being
> > > > > > > > executed, implying a total of 50 updates.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli <
> > alex@porcelli.me>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Francisco,
> > > > > > > > >
> > > > > > > > > I noticed that your PR has been merged, but I was expecting
> > (at
> > > > > least
> > > > > > > > > was my understanding from this thread) that before merging
> > some
> > > > > > > > > benchmark data would be shared in advance - to assess the
> > > > > > cost/benefit
> > > > > > > > > of such a decent size change.
> > > > > > > > >
> > > > > > > > > Do you have any information to share?
> > > > > > > > >
> > > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM Francisco Javier Tirado
> Sarti
> > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Yes, as intended, now we have one select and one
> > > insert/update
> > > > > per
> > > > > > > node
> > > > > > > > > > event.
> > > > > > > > > > I moved the PR as ready for review and give @Pere
> Fernandez
> > > > Perez
> > > > > > > > > > <pe...@redhat.com> permission to the branch so he can
> > > edit
> > > > it
> > > > > > in
> > > > > > > > the
> > > > > > > > > > next two weeks (Ill be on PTO)  if desired, before
> merging.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex Porcelli <
> > > > alex@porcelli.me>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Cool, thank you Francisco!
> > > > > > > > > > >
> > > > > > > > > > > Did you manage to get some preliminary data about
> > > > improvements?
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM Francisco Javier
> Tirado
> > > > Sarti
> > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, after some delay because of quarkus 3 migration.
> > Im
> > > > > > refining
> > > > > > > > > this
> > > > > > > > > > > > draft PR
> > > > > > > > > > > >
> > > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex Porcelli <
> > > > > > alex@porcelli.me>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Any update or new findings on this topic?
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM Francisco Javier
> > Tirado
> > > > > Sarti
> > > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > > > > After considering different options to improve
> > > > > performance,
> > > > > > > we
> > > > > > > > > feel
> > > > > > > > > > > that
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > is time to "partially" move away from the current
> > Map
> > > > > style
> > > > > > > > > > > interface (
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > > > > > > > > )
> > > > > > > > > > > > > > which was shared with Trusty, to one more
> suitable
> > > for
> > > > > > usage
> > > > > > > > > with a
> > > > > > > > > > > > > > relational DB like postgresql (but still
> compatible
> > > > with
> > > > > > big
> > > > > > > > > table
> > > > > > > > > > > dbs).
> > > > > > > > > > > > > > The idea will be to replace generic Storage
> > interface
> > > > by
> > > > > > four
> > > > > > > > > > > specific
> > > > > > > > > > > > > > interfaces (which will inherit from a common one
> > that
> > > > > keeps
> > > > > > > the
> > > > > > > > > query
> > > > > > > > > > > > > part
> > > > > > > > > > > > > > at is it. with get and query methods), that will
> > > > include
> > > > > > the
> > > > > > > > > required
> > > > > > > > > > > > > > modification operations for the four DataIndex
> > > > "domains":
> > > > > > > > > > > > > processinstance,
> > > > > > > > > > > > > > usertask, processdefinitions and jobs. Those
> > > interfaces
> > > > > > will
> > > > > > > > > define
> > > > > > > > > > > > > methods
> > > > > > > > > > > > > > like addNode, addVariable, updateTask,
> > > > addAttachment.....
> > > > > > > that
> > > > > > > > > will
> > > > > > > > > > > allow
> > > > > > > > > > > > > > the persistent layer implementation  to just
> update
> > > the
> > > > > > > needed
> > > > > > > > > info
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > DB  (for example, for addNode in Postgres, just
> > > insert
> > > > a
> > > > > > row
> > > > > > > > into
> > > > > > > > > > > nodes
> > > > > > > > > > > > > > table, for addNode in Mongo, basically the same
> > > atomic
> > > > > > upsert
> > > > > > > > > > > operation
> > > > > > > > > > > > > > that is currently done). Therefore, we increase
> > > > > performance
> > > > > > > for
> > > > > > > > > > > Postgres
> > > > > > > > > > > > > > and keep the current one for Mongo. The current
> DB
> > > > > schemas
> > > > > > > > won't
> > > > > > > > > be
> > > > > > > > > > > > > > touched.
> > > > > > > > > > > > > > Since the code change is large, I do not think
> I'll
> > > be
> > > > > able
> > > > > > > to
> > > > > > > > > have
> > > > > > > > > > > the
> > > > > > > > > > > > > PR
> > > > > > > > > > > > > > ready till next week.
> > > > > > > > > > > > > > But before starting, please let me know if that
> > > > approach
> > > > > is
> > > > > > > > fine
> > > > > > > > > for
> > > > > > > > > > > you.
> > > > > > > > > > > > > > Best regards.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 6:55 PM Alex Porcelli <
> > > > > > > > alex@porcelli.me>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thank you Francisco to getting deeper on this…
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Looking forward to see the results of your
> > > suggested
> > > > > > > > > improvements.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 9:40 AM Francisco
> Javier
> > > > Tirado
> > > > > > > > Sarti <
> > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I forgot to attach the queries
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 3:04 PM Francisco
> > Javier
> > > > > Tirado
> > > > > > > > > Sarti <
> > > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >> Hi,
> > > > > > > > > > > > > > > >> A brief update on this topic.
> > > > > > > > > > > > > > > >> After doing a simple test with example
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > > > > > > > > > ,
> > > > > > > > > > > > > > > >> the number of updates over Nodes table is
> n*n,
> > > so
> > > > we
> > > > > > > > manage
> > > > > > > > > to
> > > > > > > > > > > > > obtain a
> > > > > > > > > > > > > > > >> perfect quadratic performance degradation.
> The
> > > > > problem
> > > > > > > is
> > > > > > > > > worse
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > >> of Serverless Workflow than in BPMN because
> we
> > > the
> > > > > > > number
> > > > > > > > of
> > > > > > > > > > > nodes
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > >> greater than the number of states. In that
> > > > example N
> > > > > > is
> > > > > > > > 16,
> > > > > > > > > but
> > > > > > > > > > > for
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > >> complex workflow it would be certainly
> large.
> > > > > > > > > > > > > > > >> I think that this is more related to how we
> > are
> > > > > > handling
> > > > > > > > > JPA in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > code,
> > > > > > > > > > > > > > > >> in particular the mapping from model to
> entity
> > > > > > > (basically
> > > > > > > > > JPA is
> > > > > > > > > > > > > blind
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > >> has to update all nodes for every write
> > because
> > > it
> > > > > > > > believes
> > > > > > > > > the
> > > > > > > > > > > > > node has
> > > > > > > > > > > > > > > >> been updated, although it is not) than an
> > issue
> > > in
> > > > > the
> > > > > > > > table
> > > > > > > > > > > > > definition.
> > > > > > > > > > > > > > > >> In fact, when using JPA, separating the
> server
> > > > model
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > > > JPA
> > > > > > > > > > > > > > > entity is
> > > > > > > > > > > > > > > >> not a good idea, especially if the entity
> > > contains
> > > > > > > > > collections.
> > > > > > > > > > > I
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > try
> > > > > > > > > > > > > > > >> to change that without breaking anything.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> On Wed, Nov 22, 2023 at 12:10 PM Enrique
> > > Gonzalez
> > > > > > > > Martinez <
> > > > > > > > > > > > > > > >> egonzalez@apache.org> wrote:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >>> After the events split you now will need to
> > > > create
> > > > > a
> > > > > > > node
> > > > > > > > > > > instance
> > > > > > > > > > > > > > > >>> model instance of making independent from
> the
> > > > > process
> > > > > > > > > instance.
> > > > > > > > > > > > > > > >>> That should do the trick.
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>> Regarding deleting/inserting it was fixed
> at
> > > some
> > > > > > > point.
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>> El mar, 21 nov 2023 a las 20:22, Francisco
> > > Javier
> > > > > > > Tirado
> > > > > > > > > Sarti
> > > > > > > > > > > > > > > >>> (<ft...@redhat.com>) escribió:
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > Hi Martin,
> > > > > > > > > > > > > > > >>> > I have a task to review performance of
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > ProcessInstanceNodeDataEventMerger
> > > > > > > > > > > > > > > >>> > My idea is to reduce the number of delete
> > > > inserts
> > > > > > > when
> > > > > > > > > > > processing
> > > > > > > > > > > > > > > >>> events
> > > > > > > > > > > > > > > >>> > and try to do it incremental.
> > > > > > > > > > > > > > > >>> > That should improve performance.
> > > > > > > > > > > > > > > >>> > PS:
> > > > > > > > > > > > > > > >>> > I was planning to send an e-mail tomorrow
> > > > > > announcing
> > > > > > > > > that in
> > > > > > > > > > > > > case you
> > > > > > > > > > > > > > > >>> were
> > > > > > > > > > > > > > > >>> > already working on a fix for that. I
> assume
> > > you
> > > > > are
> > > > > > > not
> > > > > > > > > and I
> > > > > > > > > > > > > would
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > >>> > sending a PR soon.
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > On Tue, Nov 21, 2023 at 6:09 PM Martin
> > Weiler
> > > > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > wrote:
> > > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > > >>> > > I looked into the new examples using
> > > > data-index
> > > > > > > > > persistence
> > > > > > > > > > > > > addon -
> > > > > > > > > > > > > > > >>> Neus'
> > > > > > > > > > > > > > > >>> > > PR#1813 [1] for serverless and Pere's
> > > branch
> > > > > [2]
> > > > > > > for
> > > > > > > > > > > workflow
> > > > > > > > > > > > > > > (great
> > > > > > > > > > > > > > > >>> job
> > > > > > > > > > > > > > > >>> > > both!) - and they work without issues
> > using
> > > > > > single
> > > > > > > > > > > requests.
> > > > > > > > > > > > > > > >>> However, under
> > > > > > > > > > > > > > > >>> > > some load (I used 'ab' for testing
> with a
> > > > light
> > > > > > > > > > > concurrency of
> > > > > > > > > > > > > 10
> > > > > > > > > > > > > > > >>> parallel
> > > > > > > > > > > > > > > >>> > > requests) I ran into the following
> > > problems:
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > > (1) Large number of insert/delete calls
> > > (eg.
> > > > > for
> > > > > > > > tables
> > > > > > > > > > > such as
> > > > > > > > > > > > > > > >>> nodes,
> > > > > > > > > > > > > > > >>> > > definitions, etc)
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > > (2) Hibernate OptimisticLockExceptions
> /
> > > > > > > > > > > StaleStateExceptions
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > > (3) DB deadlocks
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > > (4) Error responses, slow response
> times
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > > The reason I am reaching out with this
> > > topic
> > > > > here
> > > > > > > is
> > > > > > > > to
> > > > > > > > > > > find
> > > > > > > > > > > > > out if
> > > > > > > > > > > > > > > >>> we are
> > > > > > > > > > > > > > > >>> > > aware of this issue, and if someone is
> > > > already
> > > > > > > > looking
> > > > > > > > > > > into or
> > > > > > > > > > > > > > > being
> > > > > > > > > > > > > > > >>> > > assigned to it?
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > > Thanks,
> > > > > > > > > > > > > > > >>> > > Martin
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > > [1]
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > >
> > > > > >
> https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > > > > > > > > > > > >>> > > [2]
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > > > > > > > > >>> > > To unsubscribe, e-mail:
> > > > > > > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > > > > >>> > > For additional commands, e-mail:
> > > > > > > > > dev-help@kie.apache.org
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > > > > > > > >>> To unsubscribe, e-mail:
> > > > > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > > > > >>> For additional commands, e-mail:
> > > > > > > dev-help@kie.apache.org
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > > > > > For additional commands, e-mail:
> > > > > > dev-help@kie.apache.org
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > > > > > To unsubscribe, e-mail:
> > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > > For additional commands, e-mail:
> > > dev-help@kie.apache.org
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > > > > For additional commands, e-mail:
> dev-help@kie.apache.org
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Francisco Javier Tirado Sarti <ft...@redhat.com>]

For the moment being, we will keep JPA till we exhaust all possibilities,
let's call switching from jpa to jdbc our hidden plan B ;)
I already told Martin, but in order everyone to know, just after writing
the previous email, I thought "what if Postgres is not automatically
indexing foreign keys like mysql?" and, eureka
Postgres doc
https://www.postgresql.org/docs/current/ddl-constraints.html
Mysql doc
https://dev.mysql.com/doc/refman/8.0/en/constraint-foreign-key.html
These are the relevant excerpt

*Postgresql*
*A foreign key must reference columns that either are a primary key or form
a unique constraint, or are columns from a non-partial unique index. This
means that the referenced columns always have an index to allow efficient
lookups on whether a referencing row has a match. Since a DELETE of a row
from the referenced table or an UPDATE of a referenced column will require
a scan of the referencing table for rows matching the old value, it is
often a good idea to index the referencing columns too. Because this is not
always needed, and there are many choices available on how to index, the
declaration of a foreign key constraint does not automatically create an
index on the referencing columns.*
*Mysql*
*MySQL requires that foreign key columns be indexed; if you create a table
with a foreign key constraint but no index on a given column, an index is
created. *

So I asked Martin to especially create an index for process_instance_id
column on nodes table
I think that will fix the problem detected on the thread dump.
The simpler process test to verify queries are fine still stands, though ;)


On Fri, Feb 9, 2024 at 5:10 PM Tibor Zimányi <tz...@apache.org> wrote:

> I always preferred pure JDBC over Hibernate myself, just for the sake of
> control of what is happening :) So I would not -1 that myself.
>
> Tibor
>
> Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado Sarti <
> ftirados@redhat.com>
> napísal(a):
>
> > Hi,
> > Usually I do not want to talk about work in progress because preliminary
> > conclusions are pretty volatile but, well, there are a couple of things
> > that can be concluded from the really valuable information that Martin
> > provided.
> > 1) In order to be able to determine if the number of statements is larger
> > than expected, I asked Martin to test with a simpler process definition.
> > One with just three nodes: start, script and end. The script one should
> > change just one variable. This way we can analyze if the number of
> queries
> > is the expected one. From the single log (audit was activated them) my
> > conclusion is that the number of insert/updates over processes and nodes
> > (there a lot over task, that I will prefer to skip for now, baby steps)
> is
> > the expected one.
> > 2) Analysing the thread dump, we see around 15 threads executing this
> line
> > at
> >
> >
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> > so its pretty clear the code to be optimized ;). I'm evaluating
> > possibilities within JPA/Hibernate, but I'm starting to think that it
> might
> > be better to switch to JDBC and skip hibernate. Our lives will be
> simpler,
> > especially with a schema relatively simple like ours (that will be my
> > recommendation if I was an external consultant)
> >
> > On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <tz...@apache.org>
> wrote:
> >
> > > Hi,
> > >
> > > this will be a bit off-topic. However as far as performance, I think we
> > > should think about that we have string primary keys (IDs). I would
> expect
> > > the database systems are much better with indexing numeric keys than
> > > strings. I remember from the past, when I was working with DBs, that
> > using
> > > strings as keys or indexes was a discouraged practice.
> > >
> > > Best regards,
> > > Tibor
> > >
> > > Dňa št 8. 2. 2024, 22:45 Martin Weiler <mw...@ibm.com.invalid>
> > > napísal(a):
> > >
> > > > I changed the test to use MongoDB [1] and I don't see a performance
> > > > degradation with this setup [2].
> > > >
> > > > Please keep us posted of your findings. Thanks!
> > > >
> > > > Martin
> > > >
> > > > [1]
> > > https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > > [2]
> > > >
> > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > > >
> > > > ________________________________________
> > > > From: Francisco Javier Tirado Sarti <ft...@redhat.com>
> > > > Sent: Wednesday, February 7, 2024 11:40 AM
> > > > To: dev@kie.apache.org
> > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> data-index
> > > > persistence addon
> > > >
> > > > yes, it can be index degradation because of size, but I believe (I
> > might
> > > be
> > > > wrong) the db is too small (yet) for that.
> > > > But, eventually, Postgres, when the DB is huge enough, unavoidably
> will
> > > > behave like the graphic that Martin sent.
> > > > Since I believe we are not huge enough (yet), lets rule out another
> > issue
> > > > by analysing the sql logs (I requested those to Martin offline and he
> > is
> > > > going to kindly collect them).
> > > > Also Im curious to know if Mongo behave in the same way.
> > > >
> > > > On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez Martinez <
> > > > egonzalez@apache.org> wrote:
> > > >
> > > > > Hi Francisco,
> > > > > I would highly recommend to check indexes and how the updates work
> in
> > > > data
> > > > > index to avoid full scan table and lock the full table. Some db are
> > > very
> > > > > sensitive to that.
> > > > >
> > > > > El mié, 7 feb 2024, 18:41, Francisco Javier Tirado Sarti <
> > > > > ftirados@redhat.com> escribió:
> > > > >
> > > > > > Hi Martin,
> > > > > > While I analyze the data, let me ask you if it is possible to
> > perform
> > > > > > another check (similar in a way to disabling data-index like you
> > do)
> > > > Can
> > > > > > you switch to MongoDB persistence and check if the same
> degradation
> > > > that
> > > > > is
> > > > > > there for postgres remains?
> > > > > > I do not know if this is feasible but will certainly indicate the
> > > > problem
> > > > > > is on the postgres storage layer and I do not have a clear
> > prediction
> > > > of
> > > > > > what we will see when doing this switch.
> > > > > >
> > > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler
> > <mweiler@ibm.com.invalid
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Francisco,
> > > > > > >
> > > > > > > thanks for your work on this important topic!
> > > > > > >
> > > > > > > I would like to share some test results here, which might help
> to
> > > > > improve
> > > > > > > the codebase even further. I am using the jmeter based test
> case
> > > from
> > > > > > Pere
> > > > > > > and Enrique (thanks guys!) [1] which uses a load of 30 threads
> to
> > > > > > >
> > > > > > > 1) start a new process instance (POST)
> > > > > > > 2) retrieve tasks for a user (GET)
> > > > > > > 3) fetches task details (GET)
> > > > > > > 4) complete a task (POST)
> > > > > > > 5) execute a query on data-audit
> > > > > > >
> > > > > > > With this test setup, I noticed that the performance for the
> POST
> > > > > > > requests, in particular the one to start a new process
> instance,
> > > > > degrades
> > > > > > > over time - see graph [2]. If I run the same test without
> > > data-index,
> > > > > > then
> > > > > > > there is no such performance degradation [3]. You can find a
> > thread
> > > > > dump
> > > > > > > captured a few minutes into the first test here [4] that might
> > help
> > > > to
> > > > > > see
> > > > > > > some of the contention points.
> > > > > > >
> > > > > > > I'd appreciate if you could take a look and see if there is
> > > something
> > > > > > that
> > > > > > > can be further improved based on your previous work. If you
> need
> > > any
> > > > > > > additional data, let me know, but otherwise it is
> straightforward
> > > to
> > > > > run
> > > > > > > the jmeter test as well.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Martin
> > > > > > >
> > > > > > > [1] https://github.com/pefernan/job-service-refactor-test/
> > > > > > > [2]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > > > [3]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > > > [4]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > > >
> > > > > > > ________________________________________
> > > > > > > From: Francisco Javier Tirado Sarti <ft...@redhat.com>
> > > > > > > Sent: Wednesday, January 17, 2024 9:13 AM
> > > > > > > To: dev@kie.apache.org
> > > > > > > Cc: Pere Fernandez Perez
> > > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > > > data-index
> > > > > > > persistence addon
> > > > > > >
> > > > > > > Hi Alex,
> > > > > > > I did not take times (which depends on a number of variables
> that
> > > > > > > drastically change between environments), but verify that the
> > > number
> > > > of
> > > > > > > updates has been reduced drastically without losing
> > functionality,
> > > > > which
> > > > > > is
> > > > > > > objectively a good thing. If before the change, for every node
> > > > > executed,
> > > > > > we
> > > > > > > have an update for every node previously executed, so if a
> > process
> > > > have
> > > > > > 50
> > > > > > > nodes to execute, we were performing nearly 50*51/2 updates,
> > which
> > > > > gives
> > > > > > us
> > > > > > > a total of  1275 updates, now we have just one for every node
> > being
> > > > > > > executed, implying a total of 50 updates.
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli <
> alex@porcelli.me>
> > > > > wrote:
> > > > > > >
> > > > > > > > Francisco,
> > > > > > > >
> > > > > > > > I noticed that your PR has been merged, but I was expecting
> (at
> > > > least
> > > > > > > > was my understanding from this thread) that before merging
> some
> > > > > > > > benchmark data would be shared in advance - to assess the
> > > > > cost/benefit
> > > > > > > > of such a decent size change.
> > > > > > > >
> > > > > > > > Do you have any information to share?
> > > > > > > >
> > > > > > > > On Sat, Dec 23, 2023 at 4:02 AM Francisco Javier Tirado Sarti
> > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > >
> > > > > > > > > Yes, as intended, now we have one select and one
> > insert/update
> > > > per
> > > > > > node
> > > > > > > > > event.
> > > > > > > > > I moved the PR as ready for review and give @Pere Fernandez
> > > Perez
> > > > > > > > > <pe...@redhat.com> permission to the branch so he can
> > edit
> > > it
> > > > > in
> > > > > > > the
> > > > > > > > > next two weeks (Ill be on PTO)  if desired, before merging.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex Porcelli <
> > > alex@porcelli.me>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Cool, thank you Francisco!
> > > > > > > > > >
> > > > > > > > > > Did you manage to get some preliminary data about
> > > improvements?
> > > > > > > > > >
> > > > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM Francisco Javier Tirado
> > > Sarti
> > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Yes, after some delay because of quarkus 3 migration.
> Im
> > > > > refining
> > > > > > > > this
> > > > > > > > > > > draft PR
> > > > > > > > > > >
> > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex Porcelli <
> > > > > alex@porcelli.me>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Any update or new findings on this topic?
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM Francisco Javier
> Tirado
> > > > Sarti
> > > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > > > After considering different options to improve
> > > > performance,
> > > > > > we
> > > > > > > > feel
> > > > > > > > > > that
> > > > > > > > > > > > it
> > > > > > > > > > > > > is time to "partially" move away from the current
> Map
> > > > style
> > > > > > > > > > interface (
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > > > > > > > )
> > > > > > > > > > > > > which was shared with Trusty, to one more suitable
> > for
> > > > > usage
> > > > > > > > with a
> > > > > > > > > > > > > relational DB like postgresql (but still compatible
> > > with
> > > > > big
> > > > > > > > table
> > > > > > > > > > dbs).
> > > > > > > > > > > > > The idea will be to replace generic Storage
> interface
> > > by
> > > > > four
> > > > > > > > > > specific
> > > > > > > > > > > > > interfaces (which will inherit from a common one
> that
> > > > keeps
> > > > > > the
> > > > > > > > query
> > > > > > > > > > > > part
> > > > > > > > > > > > > at is it. with get and query methods), that will
> > > include
> > > > > the
> > > > > > > > required
> > > > > > > > > > > > > modification operations for the four DataIndex
> > > "domains":
> > > > > > > > > > > > processinstance,
> > > > > > > > > > > > > usertask, processdefinitions and jobs. Those
> > interfaces
> > > > > will
> > > > > > > > define
> > > > > > > > > > > > methods
> > > > > > > > > > > > > like addNode, addVariable, updateTask,
> > > addAttachment.....
> > > > > > that
> > > > > > > > will
> > > > > > > > > > allow
> > > > > > > > > > > > > the persistent layer implementation  to just update
> > the
> > > > > > needed
> > > > > > > > info
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > DB  (for example, for addNode in Postgres, just
> > insert
> > > a
> > > > > row
> > > > > > > into
> > > > > > > > > > nodes
> > > > > > > > > > > > > table, for addNode in Mongo, basically the same
> > atomic
> > > > > upsert
> > > > > > > > > > operation
> > > > > > > > > > > > > that is currently done). Therefore, we increase
> > > > performance
> > > > > > for
> > > > > > > > > > Postgres
> > > > > > > > > > > > > and keep the current one for Mongo. The current DB
> > > > schemas
> > > > > > > won't
> > > > > > > > be
> > > > > > > > > > > > > touched.
> > > > > > > > > > > > > Since the code change is large, I do not think I'll
> > be
> > > > able
> > > > > > to
> > > > > > > > have
> > > > > > > > > > the
> > > > > > > > > > > > PR
> > > > > > > > > > > > > ready till next week.
> > > > > > > > > > > > > But before starting, please let me know if that
> > > approach
> > > > is
> > > > > > > fine
> > > > > > > > for
> > > > > > > > > > you.
> > > > > > > > > > > > > Best regards.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Nov 24, 2023 at 6:55 PM Alex Porcelli <
> > > > > > > alex@porcelli.me>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Thank you Francisco to getting deeper on this…
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Looking forward to see the results of your
> > suggested
> > > > > > > > improvements.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 9:40 AM Francisco Javier
> > > Tirado
> > > > > > > Sarti <
> > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I forgot to attach the queries
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 3:04 PM Francisco
> Javier
> > > > Tirado
> > > > > > > > Sarti <
> > > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> Hi,
> > > > > > > > > > > > > > >> A brief update on this topic.
> > > > > > > > > > > > > > >> After doing a simple test with example
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > > > > > > > > ,
> > > > > > > > > > > > > > >> the number of updates over Nodes table is n*n,
> > so
> > > we
> > > > > > > manage
> > > > > > > > to
> > > > > > > > > > > > obtain a
> > > > > > > > > > > > > > >> perfect quadratic performance degradation. The
> > > > problem
> > > > > > is
> > > > > > > > worse
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > case
> > > > > > > > > > > > > > >> of Serverless Workflow than in BPMN because we
> > the
> > > > > > number
> > > > > > > of
> > > > > > > > > > nodes
> > > > > > > > > > > > is
> > > > > > > > > > > > > > >> greater than the number of states. In that
> > > example N
> > > > > is
> > > > > > > 16,
> > > > > > > > but
> > > > > > > > > > for
> > > > > > > > > > > > a
> > > > > > > > > > > > > > more
> > > > > > > > > > > > > > >> complex workflow it would be certainly large.
> > > > > > > > > > > > > > >> I think that this is more related to how we
> are
> > > > > handling
> > > > > > > > JPA in
> > > > > > > > > > the
> > > > > > > > > > > > > > code,
> > > > > > > > > > > > > > >> in particular the mapping from model to entity
> > > > > > (basically
> > > > > > > > JPA is
> > > > > > > > > > > > blind
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > >> has to update all nodes for every write
> because
> > it
> > > > > > > believes
> > > > > > > > the
> > > > > > > > > > > > node has
> > > > > > > > > > > > > > >> been updated, although it is not) than an
> issue
> > in
> > > > the
> > > > > > > table
> > > > > > > > > > > > definition.
> > > > > > > > > > > > > > >> In fact, when using JPA, separating the server
> > > model
> > > > > > from
> > > > > > > > the
> > > > > > > > > > JPA
> > > > > > > > > > > > > > entity is
> > > > > > > > > > > > > > >> not a good idea, especially if the entity
> > contains
> > > > > > > > collections.
> > > > > > > > > > I
> > > > > > > > > > > > will
> > > > > > > > > > > > > > try
> > > > > > > > > > > > > > >> to change that without breaking anything.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> On Wed, Nov 22, 2023 at 12:10 PM Enrique
> > Gonzalez
> > > > > > > Martinez <
> > > > > > > > > > > > > > >> egonzalez@apache.org> wrote:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>> After the events split you now will need to
> > > create
> > > > a
> > > > > > node
> > > > > > > > > > instance
> > > > > > > > > > > > > > >>> model instance of making independent from the
> > > > process
> > > > > > > > instance.
> > > > > > > > > > > > > > >>> That should do the trick.
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> Regarding deleting/inserting it was fixed at
> > some
> > > > > > point.
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> El mar, 21 nov 2023 a las 20:22, Francisco
> > Javier
> > > > > > Tirado
> > > > > > > > Sarti
> > > > > > > > > > > > > > >>> (<ft...@redhat.com>) escribió:
> > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > >>> > Hi Martin,
> > > > > > > > > > > > > > >>> > I have a task to review performance of
> > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > >>> > ProcessInstanceNodeDataEventMerger
> > > > > > > > > > > > > > >>> > My idea is to reduce the number of delete
> > > inserts
> > > > > > when
> > > > > > > > > > processing
> > > > > > > > > > > > > > >>> events
> > > > > > > > > > > > > > >>> > and try to do it incremental.
> > > > > > > > > > > > > > >>> > That should improve performance.
> > > > > > > > > > > > > > >>> > PS:
> > > > > > > > > > > > > > >>> > I was planning to send an e-mail tomorrow
> > > > > announcing
> > > > > > > > that in
> > > > > > > > > > > > case you
> > > > > > > > > > > > > > >>> were
> > > > > > > > > > > > > > >>> > already working on a fix for that. I assume
> > you
> > > > are
> > > > > > not
> > > > > > > > and I
> > > > > > > > > > > > would
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > >>> > sending a PR soon.
> > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > >>> > On Tue, Nov 21, 2023 at 6:09 PM Martin
> Weiler
> > > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > >>> > wrote:
> > > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > > >>> > > I looked into the new examples using
> > > data-index
> > > > > > > > persistence
> > > > > > > > > > > > addon -
> > > > > > > > > > > > > > >>> Neus'
> > > > > > > > > > > > > > >>> > > PR#1813 [1] for serverless and Pere's
> > branch
> > > > [2]
> > > > > > for
> > > > > > > > > > workflow
> > > > > > > > > > > > > > (great
> > > > > > > > > > > > > > >>> job
> > > > > > > > > > > > > > >>> > > both!) - and they work without issues
> using
> > > > > single
> > > > > > > > > > requests.
> > > > > > > > > > > > > > >>> However, under
> > > > > > > > > > > > > > >>> > > some load (I used 'ab' for testing with a
> > > light
> > > > > > > > > > concurrency of
> > > > > > > > > > > > 10
> > > > > > > > > > > > > > >>> parallel
> > > > > > > > > > > > > > >>> > > requests) I ran into the following
> > problems:
> > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > >>> > > (1) Large number of insert/delete calls
> > (eg.
> > > > for
> > > > > > > tables
> > > > > > > > > > such as
> > > > > > > > > > > > > > >>> nodes,
> > > > > > > > > > > > > > >>> > > definitions, etc)
> > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > >>> > > (2) Hibernate OptimisticLockExceptions /
> > > > > > > > > > StaleStateExceptions
> > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > >>> > > (3) DB deadlocks
> > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > >>> > > (4) Error responses, slow response times
> > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > >>> > > The reason I am reaching out with this
> > topic
> > > > here
> > > > > > is
> > > > > > > to
> > > > > > > > > > find
> > > > > > > > > > > > out if
> > > > > > > > > > > > > > >>> we are
> > > > > > > > > > > > > > >>> > > aware of this issue, and if someone is
> > > already
> > > > > > > looking
> > > > > > > > > > into or
> > > > > > > > > > > > > > being
> > > > > > > > > > > > > > >>> > > assigned to it?
> > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > >>> > > Thanks,
> > > > > > > > > > > > > > >>> > > Martin
> > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > >>> > > [1]
> > > > > > > > > > > > > > >>>
> > > > > > > > > >
> > > > > https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > > > > > > > > > > >>> > > [2]
> > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > >
> > > > > > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > > > > > > > >>> > > To unsubscribe, e-mail:
> > > > > > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > > > >>> > > For additional commands, e-mail:
> > > > > > > > dev-help@kie.apache.org
> > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > > > > > > > >>> To unsubscribe, e-mail:
> > > > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > > > >>> For additional commands, e-mail:
> > > > > > dev-help@kie.apache.org
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >
> > > > > > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > > > > For additional commands, e-mail:
> > > > > dev-help@kie.apache.org
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > > > > > To unsubscribe, e-mail:
> dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > For additional commands, e-mail:
> > dev-help@kie.apache.org
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Tibor Zimányi <tz...@apache.org>]

I always preferred pure JDBC over Hibernate myself, just for the sake of
control of what is happening :) So I would not -1 that myself.

Tibor

Dňa pi 9. 2. 2024, 17:00 Francisco Javier Tirado Sarti <ft...@redhat.com>
napísal(a):

> Hi,
> Usually I do not want to talk about work in progress because preliminary
> conclusions are pretty volatile but, well, there are a couple of things
> that can be concluded from the really valuable information that Martin
> provided.
> 1) In order to be able to determine if the number of statements is larger
> than expected, I asked Martin to test with a simpler process definition.
> One with just three nodes: start, script and end. The script one should
> change just one variable. This way we can analyze if the number of queries
> is the expected one. From the single log (audit was activated them) my
> conclusion is that the number of insert/updates over processes and nodes
> (there a lot over task, that I will prefer to skip for now, baby steps) is
> the expected one.
> 2) Analysing the thread dump, we see around 15 threads executing this line
> at
>
> org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
> so its pretty clear the code to be optimized ;). I'm evaluating
> possibilities within JPA/Hibernate, but I'm starting to think that it might
> be better to switch to JDBC and skip hibernate. Our lives will be simpler,
> especially with a schema relatively simple like ours (that will be my
> recommendation if I was an external consultant)
>
> On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <tz...@apache.org> wrote:
>
> > Hi,
> >
> > this will be a bit off-topic. However as far as performance, I think we
> > should think about that we have string primary keys (IDs). I would expect
> > the database systems are much better with indexing numeric keys than
> > strings. I remember from the past, when I was working with DBs, that
> using
> > strings as keys or indexes was a discouraged practice.
> >
> > Best regards,
> > Tibor
> >
> > Dňa št 8. 2. 2024, 22:45 Martin Weiler <mw...@ibm.com.invalid>
> > napísal(a):
> >
> > > I changed the test to use MongoDB [1] and I don't see a performance
> > > degradation with this setup [2].
> > >
> > > Please keep us posted of your findings. Thanks!
> > >
> > > Martin
> > >
> > > [1]
> > https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > > [2]
> > >
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> > >
> > > ________________________________________
> > > From: Francisco Javier Tirado Sarti <ft...@redhat.com>
> > > Sent: Wednesday, February 7, 2024 11:40 AM
> > > To: dev@kie.apache.org
> > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with data-index
> > > persistence addon
> > >
> > > yes, it can be index degradation because of size, but I believe (I
> might
> > be
> > > wrong) the db is too small (yet) for that.
> > > But, eventually, Postgres, when the DB is huge enough, unavoidably will
> > > behave like the graphic that Martin sent.
> > > Since I believe we are not huge enough (yet), lets rule out another
> issue
> > > by analysing the sql logs (I requested those to Martin offline and he
> is
> > > going to kindly collect them).
> > > Also Im curious to know if Mongo behave in the same way.
> > >
> > > On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez Martinez <
> > > egonzalez@apache.org> wrote:
> > >
> > > > Hi Francisco,
> > > > I would highly recommend to check indexes and how the updates work in
> > > data
> > > > index to avoid full scan table and lock the full table. Some db are
> > very
> > > > sensitive to that.
> > > >
> > > > El mié, 7 feb 2024, 18:41, Francisco Javier Tirado Sarti <
> > > > ftirados@redhat.com> escribió:
> > > >
> > > > > Hi Martin,
> > > > > While I analyze the data, let me ask you if it is possible to
> perform
> > > > > another check (similar in a way to disabling data-index like you
> do)
> > > Can
> > > > > you switch to MongoDB persistence and check if the same degradation
> > > that
> > > > is
> > > > > there for postgres remains?
> > > > > I do not know if this is feasible but will certainly indicate the
> > > problem
> > > > > is on the postgres storage layer and I do not have a clear
> prediction
> > > of
> > > > > what we will see when doing this switch.
> > > > >
> > > > > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler
> <mweiler@ibm.com.invalid
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi Francisco,
> > > > > >
> > > > > > thanks for your work on this important topic!
> > > > > >
> > > > > > I would like to share some test results here, which might help to
> > > > improve
> > > > > > the codebase even further. I am using the jmeter based test case
> > from
> > > > > Pere
> > > > > > and Enrique (thanks guys!) [1] which uses a load of 30 threads to
> > > > > >
> > > > > > 1) start a new process instance (POST)
> > > > > > 2) retrieve tasks for a user (GET)
> > > > > > 3) fetches task details (GET)
> > > > > > 4) complete a task (POST)
> > > > > > 5) execute a query on data-audit
> > > > > >
> > > > > > With this test setup, I noticed that the performance for the POST
> > > > > > requests, in particular the one to start a new process instance,
> > > > degrades
> > > > > > over time - see graph [2]. If I run the same test without
> > data-index,
> > > > > then
> > > > > > there is no such performance degradation [3]. You can find a
> thread
> > > > dump
> > > > > > captured a few minutes into the first test here [4] that might
> help
> > > to
> > > > > see
> > > > > > some of the contention points.
> > > > > >
> > > > > > I'd appreciate if you could take a look and see if there is
> > something
> > > > > that
> > > > > > can be further improved based on your previous work. If you need
> > any
> > > > > > additional data, let me know, but otherwise it is straightforward
> > to
> > > > run
> > > > > > the jmeter test as well.
> > > > > >
> > > > > > Thanks,
> > > > > > Martin
> > > > > >
> > > > > > [1] https://github.com/pefernan/job-service-refactor-test/
> > > > > > [2]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > > [3]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > > [4]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > > >
> > > > > > ________________________________________
> > > > > > From: Francisco Javier Tirado Sarti <ft...@redhat.com>
> > > > > > Sent: Wednesday, January 17, 2024 9:13 AM
> > > > > > To: dev@kie.apache.org
> > > > > > Cc: Pere Fernandez Perez
> > > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > > data-index
> > > > > > persistence addon
> > > > > >
> > > > > > Hi Alex,
> > > > > > I did not take times (which depends on a number of variables that
> > > > > > drastically change between environments), but verify that the
> > number
> > > of
> > > > > > updates has been reduced drastically without losing
> functionality,
> > > > which
> > > > > is
> > > > > > objectively a good thing. If before the change, for every node
> > > > executed,
> > > > > we
> > > > > > have an update for every node previously executed, so if a
> process
> > > have
> > > > > 50
> > > > > > nodes to execute, we were performing nearly 50*51/2 updates,
> which
> > > > gives
> > > > > us
> > > > > > a total of  1275 updates, now we have just one for every node
> being
> > > > > > executed, implying a total of 50 updates.
> > > > > >
> > > > > >
> > > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli <al...@porcelli.me>
> > > > wrote:
> > > > > >
> > > > > > > Francisco,
> > > > > > >
> > > > > > > I noticed that your PR has been merged, but I was expecting (at
> > > least
> > > > > > > was my understanding from this thread) that before merging some
> > > > > > > benchmark data would be shared in advance - to assess the
> > > > cost/benefit
> > > > > > > of such a decent size change.
> > > > > > >
> > > > > > > Do you have any information to share?
> > > > > > >
> > > > > > > On Sat, Dec 23, 2023 at 4:02 AM Francisco Javier Tirado Sarti
> > > > > > > <ft...@redhat.com> wrote:
> > > > > > > >
> > > > > > > > Yes, as intended, now we have one select and one
> insert/update
> > > per
> > > > > node
> > > > > > > > event.
> > > > > > > > I moved the PR as ready for review and give @Pere Fernandez
> > Perez
> > > > > > > > <pe...@redhat.com> permission to the branch so he can
> edit
> > it
> > > > in
> > > > > > the
> > > > > > > > next two weeks (Ill be on PTO)  if desired, before merging.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex Porcelli <
> > alex@porcelli.me>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Cool, thank you Francisco!
> > > > > > > > >
> > > > > > > > > Did you manage to get some preliminary data about
> > improvements?
> > > > > > > > >
> > > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM Francisco Javier Tirado
> > Sarti
> > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Yes, after some delay because of quarkus 3 migration. Im
> > > > refining
> > > > > > > this
> > > > > > > > > > draft PR
> > > > > > > > > >
> > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > > > > >
> > > > > > > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex Porcelli <
> > > > alex@porcelli.me>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Any update or new findings on this topic?
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM Francisco Javier Tirado
> > > Sarti
> > > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Alex,
> > > > > > > > > > > > After considering different options to improve
> > > performance,
> > > > > we
> > > > > > > feel
> > > > > > > > > that
> > > > > > > > > > > it
> > > > > > > > > > > > is time to "partially" move away from the current Map
> > > style
> > > > > > > > > interface (
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > > > > > > )
> > > > > > > > > > > > which was shared with Trusty, to one more suitable
> for
> > > > usage
> > > > > > > with a
> > > > > > > > > > > > relational DB like postgresql (but still compatible
> > with
> > > > big
> > > > > > > table
> > > > > > > > > dbs).
> > > > > > > > > > > > The idea will be to replace generic Storage interface
> > by
> > > > four
> > > > > > > > > specific
> > > > > > > > > > > > interfaces (which will inherit from a common one that
> > > keeps
> > > > > the
> > > > > > > query
> > > > > > > > > > > part
> > > > > > > > > > > > at is it. with get and query methods), that will
> > include
> > > > the
> > > > > > > required
> > > > > > > > > > > > modification operations for the four DataIndex
> > "domains":
> > > > > > > > > > > processinstance,
> > > > > > > > > > > > usertask, processdefinitions and jobs. Those
> interfaces
> > > > will
> > > > > > > define
> > > > > > > > > > > methods
> > > > > > > > > > > > like addNode, addVariable, updateTask,
> > addAttachment.....
> > > > > that
> > > > > > > will
> > > > > > > > > allow
> > > > > > > > > > > > the persistent layer implementation  to just update
> the
> > > > > needed
> > > > > > > info
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > DB  (for example, for addNode in Postgres, just
> insert
> > a
> > > > row
> > > > > > into
> > > > > > > > > nodes
> > > > > > > > > > > > table, for addNode in Mongo, basically the same
> atomic
> > > > upsert
> > > > > > > > > operation
> > > > > > > > > > > > that is currently done). Therefore, we increase
> > > performance
> > > > > for
> > > > > > > > > Postgres
> > > > > > > > > > > > and keep the current one for Mongo. The current DB
> > > schemas
> > > > > > won't
> > > > > > > be
> > > > > > > > > > > > touched.
> > > > > > > > > > > > Since the code change is large, I do not think I'll
> be
> > > able
> > > > > to
> > > > > > > have
> > > > > > > > > the
> > > > > > > > > > > PR
> > > > > > > > > > > > ready till next week.
> > > > > > > > > > > > But before starting, please let me know if that
> > approach
> > > is
> > > > > > fine
> > > > > > > for
> > > > > > > > > you.
> > > > > > > > > > > > Best regards.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Nov 24, 2023 at 6:55 PM Alex Porcelli <
> > > > > > alex@porcelli.me>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Thank you Francisco to getting deeper on this…
> > > > > > > > > > > > >
> > > > > > > > > > > > > Looking forward to see the results of your
> suggested
> > > > > > > improvements.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Nov 24, 2023 at 9:40 AM Francisco Javier
> > Tirado
> > > > > > Sarti <
> > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I forgot to attach the queries
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Nov 24, 2023 at 3:04 PM Francisco Javier
> > > Tirado
> > > > > > > Sarti <
> > > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> Hi,
> > > > > > > > > > > > > >> A brief update on this topic.
> > > > > > > > > > > > > >> After doing a simple test with example
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > > > > > > > ,
> > > > > > > > > > > > > >> the number of updates over Nodes table is n*n,
> so
> > we
> > > > > > manage
> > > > > > > to
> > > > > > > > > > > obtain a
> > > > > > > > > > > > > >> perfect quadratic performance degradation. The
> > > problem
> > > > > is
> > > > > > > worse
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > case
> > > > > > > > > > > > > >> of Serverless Workflow than in BPMN because we
> the
> > > > > number
> > > > > > of
> > > > > > > > > nodes
> > > > > > > > > > > is
> > > > > > > > > > > > > >> greater than the number of states. In that
> > example N
> > > > is
> > > > > > 16,
> > > > > > > but
> > > > > > > > > for
> > > > > > > > > > > a
> > > > > > > > > > > > > more
> > > > > > > > > > > > > >> complex workflow it would be certainly large.
> > > > > > > > > > > > > >> I think that this is more related to how we are
> > > > handling
> > > > > > > JPA in
> > > > > > > > > the
> > > > > > > > > > > > > code,
> > > > > > > > > > > > > >> in particular the mapping from model to entity
> > > > > (basically
> > > > > > > JPA is
> > > > > > > > > > > blind
> > > > > > > > > > > > > and
> > > > > > > > > > > > > >> has to update all nodes for every write because
> it
> > > > > > believes
> > > > > > > the
> > > > > > > > > > > node has
> > > > > > > > > > > > > >> been updated, although it is not) than an issue
> in
> > > the
> > > > > > table
> > > > > > > > > > > definition.
> > > > > > > > > > > > > >> In fact, when using JPA, separating the server
> > model
> > > > > from
> > > > > > > the
> > > > > > > > > JPA
> > > > > > > > > > > > > entity is
> > > > > > > > > > > > > >> not a good idea, especially if the entity
> contains
> > > > > > > collections.
> > > > > > > > > I
> > > > > > > > > > > will
> > > > > > > > > > > > > try
> > > > > > > > > > > > > >> to change that without breaking anything.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> On Wed, Nov 22, 2023 at 12:10 PM Enrique
> Gonzalez
> > > > > > Martinez <
> > > > > > > > > > > > > >> egonzalez@apache.org> wrote:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>> After the events split you now will need to
> > create
> > > a
> > > > > node
> > > > > > > > > instance
> > > > > > > > > > > > > >>> model instance of making independent from the
> > > process
> > > > > > > instance.
> > > > > > > > > > > > > >>> That should do the trick.
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> Regarding deleting/inserting it was fixed at
> some
> > > > > point.
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> El mar, 21 nov 2023 a las 20:22, Francisco
> Javier
> > > > > Tirado
> > > > > > > Sarti
> > > > > > > > > > > > > >>> (<ft...@redhat.com>) escribió:
> > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > >>> > Hi Martin,
> > > > > > > > > > > > > >>> > I have a task to review performance of
> > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > >>> > ProcessInstanceNodeDataEventMerger
> > > > > > > > > > > > > >>> > My idea is to reduce the number of delete
> > inserts
> > > > > when
> > > > > > > > > processing
> > > > > > > > > > > > > >>> events
> > > > > > > > > > > > > >>> > and try to do it incremental.
> > > > > > > > > > > > > >>> > That should improve performance.
> > > > > > > > > > > > > >>> > PS:
> > > > > > > > > > > > > >>> > I was planning to send an e-mail tomorrow
> > > > announcing
> > > > > > > that in
> > > > > > > > > > > case you
> > > > > > > > > > > > > >>> were
> > > > > > > > > > > > > >>> > already working on a fix for that. I assume
> you
> > > are
> > > > > not
> > > > > > > and I
> > > > > > > > > > > would
> > > > > > > > > > > > > be
> > > > > > > > > > > > > >>> > sending a PR soon.
> > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > >>> > On Tue, Nov 21, 2023 at 6:09 PM Martin Weiler
> > > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > >>> > wrote:
> > > > > > > > > > > > > >>> >
> > > > > > > > > > > > > >>> > > I looked into the new examples using
> > data-index
> > > > > > > persistence
> > > > > > > > > > > addon -
> > > > > > > > > > > > > >>> Neus'
> > > > > > > > > > > > > >>> > > PR#1813 [1] for serverless and Pere's
> branch
> > > [2]
> > > > > for
> > > > > > > > > workflow
> > > > > > > > > > > > > (great
> > > > > > > > > > > > > >>> job
> > > > > > > > > > > > > >>> > > both!) - and they work without issues using
> > > > single
> > > > > > > > > requests.
> > > > > > > > > > > > > >>> However, under
> > > > > > > > > > > > > >>> > > some load (I used 'ab' for testing with a
> > light
> > > > > > > > > concurrency of
> > > > > > > > > > > 10
> > > > > > > > > > > > > >>> parallel
> > > > > > > > > > > > > >>> > > requests) I ran into the following
> problems:
> > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > >>> > > (1) Large number of insert/delete calls
> (eg.
> > > for
> > > > > > tables
> > > > > > > > > such as
> > > > > > > > > > > > > >>> nodes,
> > > > > > > > > > > > > >>> > > definitions, etc)
> > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > >>> > > (2) Hibernate OptimisticLockExceptions /
> > > > > > > > > StaleStateExceptions
> > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > >>> > > (3) DB deadlocks
> > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > >>> > > (4) Error responses, slow response times
> > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > >>> > > The reason I am reaching out with this
> topic
> > > here
> > > > > is
> > > > > > to
> > > > > > > > > find
> > > > > > > > > > > out if
> > > > > > > > > > > > > >>> we are
> > > > > > > > > > > > > >>> > > aware of this issue, and if someone is
> > already
> > > > > > looking
> > > > > > > > > into or
> > > > > > > > > > > > > being
> > > > > > > > > > > > > >>> > > assigned to it?
> > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > >>> > > Thanks,
> > > > > > > > > > > > > >>> > > Martin
> > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > >>> > > [1]
> > > > > > > > > > > > > >>>
> > > > > > > > >
> > > > https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > > > > > > > > > >>> > > [2]
> > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > > > > > >>> > > To unsubscribe, e-mail:
> > > > > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > > >>> > > For additional commands, e-mail:
> > > > > > > dev-help@kie.apache.org
> > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > >>> > >
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>>
> > > > > > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > > > > > > >>> To unsubscribe, e-mail:
> > > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > > >>> For additional commands, e-mail:
> > > > > dev-help@kie.apache.org
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > > > > > > To unsubscribe, e-mail:
> > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > > > For additional commands, e-mail:
> > > > dev-help@kie.apache.org
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > > > > For additional commands, e-mail:
> dev-help@kie.apache.org
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Francisco Javier Tirado Sarti <ft...@redhat.com>]

Hi,
Usually I do not want to talk about work in progress because preliminary
conclusions are pretty volatile but, well, there are a couple of things
that can be concluded from the really valuable information that Martin
provided.
1) In order to be able to determine if the number of statements is larger
than expected, I asked Martin to test with a simpler process definition.
One with just three nodes: start, script and end. The script one should
change just one variable. This way we can analyze if the number of queries
is the expected one. From the single log (audit was activated them) my
conclusion is that the number of insert/updates over processes and nodes
(there a lot over task, that I will prefer to skip for now, baby steps) is
the expected one.
2) Analysing the thread dump, we see around 15 threads executing this line
at
org.kie.kogito.index.jpa.storage.ProcessInstanceEntityStorage.indexNode(ProcessInstanceEntityStorage.java:125),
so its pretty clear the code to be optimized ;). I'm evaluating
possibilities within JPA/Hibernate, but I'm starting to think that it might
be better to switch to JDBC and skip hibernate. Our lives will be simpler,
especially with a schema relatively simple like ours (that will be my
recommendation if I was an external consultant)

On Fri, Feb 9, 2024 at 4:15 PM Tibor Zimányi <tz...@apache.org> wrote:

> Hi,
>
> this will be a bit off-topic. However as far as performance, I think we
> should think about that we have string primary keys (IDs). I would expect
> the database systems are much better with indexing numeric keys than
> strings. I remember from the past, when I was working with DBs, that using
> strings as keys or indexes was a discouraged practice.
>
> Best regards,
> Tibor
>
> Dňa št 8. 2. 2024, 22:45 Martin Weiler <mw...@ibm.com.invalid>
> napísal(a):
>
> > I changed the test to use MongoDB [1] and I don't see a performance
> > degradation with this setup [2].
> >
> > Please keep us posted of your findings. Thanks!
> >
> > Martin
> >
> > [1]
> https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> > [2]
> >
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
> >
> > ________________________________________
> > From: Francisco Javier Tirado Sarti <ft...@redhat.com>
> > Sent: Wednesday, February 7, 2024 11:40 AM
> > To: dev@kie.apache.org
> > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with data-index
> > persistence addon
> >
> > yes, it can be index degradation because of size, but I believe (I might
> be
> > wrong) the db is too small (yet) for that.
> > But, eventually, Postgres, when the DB is huge enough, unavoidably will
> > behave like the graphic that Martin sent.
> > Since I believe we are not huge enough (yet), lets rule out another issue
> > by analysing the sql logs (I requested those to Martin offline and he is
> > going to kindly collect them).
> > Also Im curious to know if Mongo behave in the same way.
> >
> > On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez Martinez <
> > egonzalez@apache.org> wrote:
> >
> > > Hi Francisco,
> > > I would highly recommend to check indexes and how the updates work in
> > data
> > > index to avoid full scan table and lock the full table. Some db are
> very
> > > sensitive to that.
> > >
> > > El mié, 7 feb 2024, 18:41, Francisco Javier Tirado Sarti <
> > > ftirados@redhat.com> escribió:
> > >
> > > > Hi Martin,
> > > > While I analyze the data, let me ask you if it is possible to perform
> > > > another check (similar in a way to disabling data-index like you do)
> > Can
> > > > you switch to MongoDB persistence and check if the same degradation
> > that
> > > is
> > > > there for postgres remains?
> > > > I do not know if this is feasible but will certainly indicate the
> > problem
> > > > is on the postgres storage layer and I do not have a clear prediction
> > of
> > > > what we will see when doing this switch.
> > > >
> > > > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler <mweiler@ibm.com.invalid
> >
> > > > wrote:
> > > >
> > > > > Hi Francisco,
> > > > >
> > > > > thanks for your work on this important topic!
> > > > >
> > > > > I would like to share some test results here, which might help to
> > > improve
> > > > > the codebase even further. I am using the jmeter based test case
> from
> > > > Pere
> > > > > and Enrique (thanks guys!) [1] which uses a load of 30 threads to
> > > > >
> > > > > 1) start a new process instance (POST)
> > > > > 2) retrieve tasks for a user (GET)
> > > > > 3) fetches task details (GET)
> > > > > 4) complete a task (POST)
> > > > > 5) execute a query on data-audit
> > > > >
> > > > > With this test setup, I noticed that the performance for the POST
> > > > > requests, in particular the one to start a new process instance,
> > > degrades
> > > > > over time - see graph [2]. If I run the same test without
> data-index,
> > > > then
> > > > > there is no such performance degradation [3]. You can find a thread
> > > dump
> > > > > captured a few minutes into the first test here [4] that might help
> > to
> > > > see
> > > > > some of the contention points.
> > > > >
> > > > > I'd appreciate if you could take a look and see if there is
> something
> > > > that
> > > > > can be further improved based on your previous work. If you need
> any
> > > > > additional data, let me know, but otherwise it is straightforward
> to
> > > run
> > > > > the jmeter test as well.
> > > > >
> > > > > Thanks,
> > > > > Martin
> > > > >
> > > > > [1] https://github.com/pefernan/job-service-refactor-test/
> > > > > [2]
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > > [3]
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > > [4]
> > > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > > >
> > > > > ________________________________________
> > > > > From: Francisco Javier Tirado Sarti <ft...@redhat.com>
> > > > > Sent: Wednesday, January 17, 2024 9:13 AM
> > > > > To: dev@kie.apache.org
> > > > > Cc: Pere Fernandez Perez
> > > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> > data-index
> > > > > persistence addon
> > > > >
> > > > > Hi Alex,
> > > > > I did not take times (which depends on a number of variables that
> > > > > drastically change between environments), but verify that the
> number
> > of
> > > > > updates has been reduced drastically without losing functionality,
> > > which
> > > > is
> > > > > objectively a good thing. If before the change, for every node
> > > executed,
> > > > we
> > > > > have an update for every node previously executed, so if a process
> > have
> > > > 50
> > > > > nodes to execute, we were performing nearly 50*51/2 updates, which
> > > gives
> > > > us
> > > > > a total of  1275 updates, now we have just one for every node being
> > > > > executed, implying a total of 50 updates.
> > > > >
> > > > >
> > > > > On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli <al...@porcelli.me>
> > > wrote:
> > > > >
> > > > > > Francisco,
> > > > > >
> > > > > > I noticed that your PR has been merged, but I was expecting (at
> > least
> > > > > > was my understanding from this thread) that before merging some
> > > > > > benchmark data would be shared in advance - to assess the
> > > cost/benefit
> > > > > > of such a decent size change.
> > > > > >
> > > > > > Do you have any information to share?
> > > > > >
> > > > > > On Sat, Dec 23, 2023 at 4:02 AM Francisco Javier Tirado Sarti
> > > > > > <ft...@redhat.com> wrote:
> > > > > > >
> > > > > > > Yes, as intended, now we have one select and one insert/update
> > per
> > > > node
> > > > > > > event.
> > > > > > > I moved the PR as ready for review and give @Pere Fernandez
> Perez
> > > > > > > <pe...@redhat.com> permission to the branch so he can edit
> it
> > > in
> > > > > the
> > > > > > > next two weeks (Ill be on PTO)  if desired, before merging.
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex Porcelli <
> alex@porcelli.me>
> > > > > wrote:
> > > > > > >
> > > > > > > > Cool, thank you Francisco!
> > > > > > > >
> > > > > > > > Did you manage to get some preliminary data about
> improvements?
> > > > > > > >
> > > > > > > > On Thu, Dec 21, 2023 at 11:52 AM Francisco Javier Tirado
> Sarti
> > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > >
> > > > > > > > > Yes, after some delay because of quarkus 3 migration. Im
> > > refining
> > > > > > this
> > > > > > > > > draft PR
> > > > > > > > >
> > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > > > >
> > > > > > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex Porcelli <
> > > alex@porcelli.me>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Any update or new findings on this topic?
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM Francisco Javier Tirado
> > Sarti
> > > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Alex,
> > > > > > > > > > > After considering different options to improve
> > performance,
> > > > we
> > > > > > feel
> > > > > > > > that
> > > > > > > > > > it
> > > > > > > > > > > is time to "partially" move away from the current Map
> > style
> > > > > > > > interface (
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > > > > > )
> > > > > > > > > > > which was shared with Trusty, to one more suitable for
> > > usage
> > > > > > with a
> > > > > > > > > > > relational DB like postgresql (but still compatible
> with
> > > big
> > > > > > table
> > > > > > > > dbs).
> > > > > > > > > > > The idea will be to replace generic Storage interface
> by
> > > four
> > > > > > > > specific
> > > > > > > > > > > interfaces (which will inherit from a common one that
> > keeps
> > > > the
> > > > > > query
> > > > > > > > > > part
> > > > > > > > > > > at is it. with get and query methods), that will
> include
> > > the
> > > > > > required
> > > > > > > > > > > modification operations for the four DataIndex
> "domains":
> > > > > > > > > > processinstance,
> > > > > > > > > > > usertask, processdefinitions and jobs. Those interfaces
> > > will
> > > > > > define
> > > > > > > > > > methods
> > > > > > > > > > > like addNode, addVariable, updateTask,
> addAttachment.....
> > > > that
> > > > > > will
> > > > > > > > allow
> > > > > > > > > > > the persistent layer implementation  to just update the
> > > > needed
> > > > > > info
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > DB  (for example, for addNode in Postgres, just insert
> a
> > > row
> > > > > into
> > > > > > > > nodes
> > > > > > > > > > > table, for addNode in Mongo, basically the same atomic
> > > upsert
> > > > > > > > operation
> > > > > > > > > > > that is currently done). Therefore, we increase
> > performance
> > > > for
> > > > > > > > Postgres
> > > > > > > > > > > and keep the current one for Mongo. The current DB
> > schemas
> > > > > won't
> > > > > > be
> > > > > > > > > > > touched.
> > > > > > > > > > > Since the code change is large, I do not think I'll be
> > able
> > > > to
> > > > > > have
> > > > > > > > the
> > > > > > > > > > PR
> > > > > > > > > > > ready till next week.
> > > > > > > > > > > But before starting, please let me know if that
> approach
> > is
> > > > > fine
> > > > > > for
> > > > > > > > you.
> > > > > > > > > > > Best regards.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Nov 24, 2023 at 6:55 PM Alex Porcelli <
> > > > > alex@porcelli.me>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thank you Francisco to getting deeper on this…
> > > > > > > > > > > >
> > > > > > > > > > > > Looking forward to see the results of your suggested
> > > > > > improvements.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Nov 24, 2023 at 9:40 AM Francisco Javier
> Tirado
> > > > > Sarti <
> > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > I forgot to attach the queries
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Nov 24, 2023 at 3:04 PM Francisco Javier
> > Tirado
> > > > > > Sarti <
> > > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Hi,
> > > > > > > > > > > > >> A brief update on this topic.
> > > > > > > > > > > > >> After doing a simple test with example
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > > > > > > ,
> > > > > > > > > > > > >> the number of updates over Nodes table is n*n, so
> we
> > > > > manage
> > > > > > to
> > > > > > > > > > obtain a
> > > > > > > > > > > > >> perfect quadratic performance degradation. The
> > problem
> > > > is
> > > > > > worse
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > case
> > > > > > > > > > > > >> of Serverless Workflow than in BPMN because we the
> > > > number
> > > > > of
> > > > > > > > nodes
> > > > > > > > > > is
> > > > > > > > > > > > >> greater than the number of states. In that
> example N
> > > is
> > > > > 16,
> > > > > > but
> > > > > > > > for
> > > > > > > > > > a
> > > > > > > > > > > > more
> > > > > > > > > > > > >> complex workflow it would be certainly large.
> > > > > > > > > > > > >> I think that this is more related to how we are
> > > handling
> > > > > > JPA in
> > > > > > > > the
> > > > > > > > > > > > code,
> > > > > > > > > > > > >> in particular the mapping from model to entity
> > > > (basically
> > > > > > JPA is
> > > > > > > > > > blind
> > > > > > > > > > > > and
> > > > > > > > > > > > >> has to update all nodes for every write because it
> > > > > believes
> > > > > > the
> > > > > > > > > > node has
> > > > > > > > > > > > >> been updated, although it is not) than an issue in
> > the
> > > > > table
> > > > > > > > > > definition.
> > > > > > > > > > > > >> In fact, when using JPA, separating the server
> model
> > > > from
> > > > > > the
> > > > > > > > JPA
> > > > > > > > > > > > entity is
> > > > > > > > > > > > >> not a good idea, especially if the entity contains
> > > > > > collections.
> > > > > > > > I
> > > > > > > > > > will
> > > > > > > > > > > > try
> > > > > > > > > > > > >> to change that without breaking anything.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> On Wed, Nov 22, 2023 at 12:10 PM Enrique Gonzalez
> > > > > Martinez <
> > > > > > > > > > > > >> egonzalez@apache.org> wrote:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>> After the events split you now will need to
> create
> > a
> > > > node
> > > > > > > > instance
> > > > > > > > > > > > >>> model instance of making independent from the
> > process
> > > > > > instance.
> > > > > > > > > > > > >>> That should do the trick.
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> Regarding deleting/inserting it was fixed at some
> > > > point.
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> El mar, 21 nov 2023 a las 20:22, Francisco Javier
> > > > Tirado
> > > > > > Sarti
> > > > > > > > > > > > >>> (<ft...@redhat.com>) escribió:
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > Hi Martin,
> > > > > > > > > > > > >>> > I have a task to review performance of
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > ProcessInstanceNodeDataEventMerger
> > > > > > > > > > > > >>> > My idea is to reduce the number of delete
> inserts
> > > > when
> > > > > > > > processing
> > > > > > > > > > > > >>> events
> > > > > > > > > > > > >>> > and try to do it incremental.
> > > > > > > > > > > > >>> > That should improve performance.
> > > > > > > > > > > > >>> > PS:
> > > > > > > > > > > > >>> > I was planning to send an e-mail tomorrow
> > > announcing
> > > > > > that in
> > > > > > > > > > case you
> > > > > > > > > > > > >>> were
> > > > > > > > > > > > >>> > already working on a fix for that. I assume you
> > are
> > > > not
> > > > > > and I
> > > > > > > > > > would
> > > > > > > > > > > > be
> > > > > > > > > > > > >>> > sending a PR soon.
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > On Tue, Nov 21, 2023 at 6:09 PM Martin Weiler
> > > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > wrote:
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > > I looked into the new examples using
> data-index
> > > > > > persistence
> > > > > > > > > > addon -
> > > > > > > > > > > > >>> Neus'
> > > > > > > > > > > > >>> > > PR#1813 [1] for serverless and Pere's branch
> > [2]
> > > > for
> > > > > > > > workflow
> > > > > > > > > > > > (great
> > > > > > > > > > > > >>> job
> > > > > > > > > > > > >>> > > both!) - and they work without issues using
> > > single
> > > > > > > > requests.
> > > > > > > > > > > > >>> However, under
> > > > > > > > > > > > >>> > > some load (I used 'ab' for testing with a
> light
> > > > > > > > concurrency of
> > > > > > > > > > 10
> > > > > > > > > > > > >>> parallel
> > > > > > > > > > > > >>> > > requests) I ran into the following problems:
> > > > > > > > > > > > >>> > >
> > > > > > > > > > > > >>> > > (1) Large number of insert/delete calls (eg.
> > for
> > > > > tables
> > > > > > > > such as
> > > > > > > > > > > > >>> nodes,
> > > > > > > > > > > > >>> > > definitions, etc)
> > > > > > > > > > > > >>> > >
> > > > > > > > > > > > >>> > > (2) Hibernate OptimisticLockExceptions /
> > > > > > > > StaleStateExceptions
> > > > > > > > > > > > >>> > >
> > > > > > > > > > > > >>> > > (3) DB deadlocks
> > > > > > > > > > > > >>> > >
> > > > > > > > > > > > >>> > > (4) Error responses, slow response times
> > > > > > > > > > > > >>> > >
> > > > > > > > > > > > >>> > > The reason I am reaching out with this topic
> > here
> > > > is
> > > > > to
> > > > > > > > find
> > > > > > > > > > out if
> > > > > > > > > > > > >>> we are
> > > > > > > > > > > > >>> > > aware of this issue, and if someone is
> already
> > > > > looking
> > > > > > > > into or
> > > > > > > > > > > > being
> > > > > > > > > > > > >>> > > assigned to it?
> > > > > > > > > > > > >>> > >
> > > > > > > > > > > > >>> > > Thanks,
> > > > > > > > > > > > >>> > > Martin
> > > > > > > > > > > > >>> > >
> > > > > > > > > > > > >>> > > [1]
> > > > > > > > > > > > >>>
> > > > > > > >
> > > https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > > > > > > > > >>> > > [2]
> > > > > > > > > > > > >>> > >
> > > > > > > > > > > > >>>
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > > > > > > > > >>> > >
> > > > > > > > > > > > >>> > >
> > > > > > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > > > > > >>> > > To unsubscribe, e-mail:
> > > > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > >>> > > For additional commands, e-mail:
> > > > > > dev-help@kie.apache.org
> > > > > > > > > > > > >>> > >
> > > > > > > > > > > > >>> > >
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>>
> > > > > > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > > > > > >>> To unsubscribe, e-mail:
> > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > >>> For additional commands, e-mail:
> > > > dev-help@kie.apache.org
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > > > > > > To unsubscribe, e-mail:
> > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > > For additional commands, e-mail:
> > > dev-help@kie.apache.org
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > >
> > > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Tibor Zimányi <tz...@apache.org>]

Hi,

this will be a bit off-topic. However as far as performance, I think we
should think about that we have string primary keys (IDs). I would expect
the database systems are much better with indexing numeric keys than
strings. I remember from the past, when I was working with DBs, that using
strings as keys or indexes was a discouraged practice.

Best regards,
Tibor

Dňa št 8. 2. 2024, 22:45 Martin Weiler <mw...@ibm.com.invalid> napísal(a):

> I changed the test to use MongoDB [1] and I don't see a performance
> degradation with this setup [2].
>
> Please keep us posted of your findings. Thanks!
>
> Martin
>
> [1] https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
> [2]
> https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing
>
> ________________________________________
> From: Francisco Javier Tirado Sarti <ft...@redhat.com>
> Sent: Wednesday, February 7, 2024 11:40 AM
> To: dev@kie.apache.org
> Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with data-index
> persistence addon
>
> yes, it can be index degradation because of size, but I believe (I might be
> wrong) the db is too small (yet) for that.
> But, eventually, Postgres, when the DB is huge enough, unavoidably will
> behave like the graphic that Martin sent.
> Since I believe we are not huge enough (yet), lets rule out another issue
> by analysing the sql logs (I requested those to Martin offline and he is
> going to kindly collect them).
> Also Im curious to know if Mongo behave in the same way.
>
> On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez Martinez <
> egonzalez@apache.org> wrote:
>
> > Hi Francisco,
> > I would highly recommend to check indexes and how the updates work in
> data
> > index to avoid full scan table and lock the full table. Some db are very
> > sensitive to that.
> >
> > El mié, 7 feb 2024, 18:41, Francisco Javier Tirado Sarti <
> > ftirados@redhat.com> escribió:
> >
> > > Hi Martin,
> > > While I analyze the data, let me ask you if it is possible to perform
> > > another check (similar in a way to disabling data-index like you do)
> Can
> > > you switch to MongoDB persistence and check if the same degradation
> that
> > is
> > > there for postgres remains?
> > > I do not know if this is feasible but will certainly indicate the
> problem
> > > is on the postgres storage layer and I do not have a clear prediction
> of
> > > what we will see when doing this switch.
> > >
> > > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler <mw...@ibm.com.invalid>
> > > wrote:
> > >
> > > > Hi Francisco,
> > > >
> > > > thanks for your work on this important topic!
> > > >
> > > > I would like to share some test results here, which might help to
> > improve
> > > > the codebase even further. I am using the jmeter based test case from
> > > Pere
> > > > and Enrique (thanks guys!) [1] which uses a load of 30 threads to
> > > >
> > > > 1) start a new process instance (POST)
> > > > 2) retrieve tasks for a user (GET)
> > > > 3) fetches task details (GET)
> > > > 4) complete a task (POST)
> > > > 5) execute a query on data-audit
> > > >
> > > > With this test setup, I noticed that the performance for the POST
> > > > requests, in particular the one to start a new process instance,
> > degrades
> > > > over time - see graph [2]. If I run the same test without data-index,
> > > then
> > > > there is no such performance degradation [3]. You can find a thread
> > dump
> > > > captured a few minutes into the first test here [4] that might help
> to
> > > see
> > > > some of the contention points.
> > > >
> > > > I'd appreciate if you could take a look and see if there is something
> > > that
> > > > can be further improved based on your previous work. If you need any
> > > > additional data, let me know, but otherwise it is straightforward to
> > run
> > > > the jmeter test as well.
> > > >
> > > > Thanks,
> > > > Martin
> > > >
> > > > [1] https://github.com/pefernan/job-service-refactor-test/
> > > > [2]
> > > >
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > > [3]
> > > >
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > > [4]
> > > >
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > > >
> > > > ________________________________________
> > > > From: Francisco Javier Tirado Sarti <ft...@redhat.com>
> > > > Sent: Wednesday, January 17, 2024 9:13 AM
> > > > To: dev@kie.apache.org
> > > > Cc: Pere Fernandez Perez
> > > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with
> data-index
> > > > persistence addon
> > > >
> > > > Hi Alex,
> > > > I did not take times (which depends on a number of variables that
> > > > drastically change between environments), but verify that the number
> of
> > > > updates has been reduced drastically without losing functionality,
> > which
> > > is
> > > > objectively a good thing. If before the change, for every node
> > executed,
> > > we
> > > > have an update for every node previously executed, so if a process
> have
> > > 50
> > > > nodes to execute, we were performing nearly 50*51/2 updates, which
> > gives
> > > us
> > > > a total of  1275 updates, now we have just one for every node being
> > > > executed, implying a total of 50 updates.
> > > >
> > > >
> > > > On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli <al...@porcelli.me>
> > wrote:
> > > >
> > > > > Francisco,
> > > > >
> > > > > I noticed that your PR has been merged, but I was expecting (at
> least
> > > > > was my understanding from this thread) that before merging some
> > > > > benchmark data would be shared in advance - to assess the
> > cost/benefit
> > > > > of such a decent size change.
> > > > >
> > > > > Do you have any information to share?
> > > > >
> > > > > On Sat, Dec 23, 2023 at 4:02 AM Francisco Javier Tirado Sarti
> > > > > <ft...@redhat.com> wrote:
> > > > > >
> > > > > > Yes, as intended, now we have one select and one insert/update
> per
> > > node
> > > > > > event.
> > > > > > I moved the PR as ready for review and give @Pere Fernandez Perez
> > > > > > <pe...@redhat.com> permission to the branch so he can edit it
> > in
> > > > the
> > > > > > next two weeks (Ill be on PTO)  if desired, before merging.
> > > > > >
> > > > > >
> > > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex Porcelli <al...@porcelli.me>
> > > > wrote:
> > > > > >
> > > > > > > Cool, thank you Francisco!
> > > > > > >
> > > > > > > Did you manage to get some preliminary data about improvements?
> > > > > > >
> > > > > > > On Thu, Dec 21, 2023 at 11:52 AM Francisco Javier Tirado Sarti
> > > > > > > <ft...@redhat.com> wrote:
> > > > > > > >
> > > > > > > > Yes, after some delay because of quarkus 3 migration. Im
> > refining
> > > > > this
> > > > > > > > draft PR
> > > > > > > >
> https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > > >
> > > > > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex Porcelli <
> > alex@porcelli.me>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Any update or new findings on this topic?
> > > > > > > > >
> > > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM Francisco Javier Tirado
> Sarti
> > > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Alex,
> > > > > > > > > > After considering different options to improve
> performance,
> > > we
> > > > > feel
> > > > > > > that
> > > > > > > > > it
> > > > > > > > > > is time to "partially" move away from the current Map
> style
> > > > > > > interface (
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > > > > )
> > > > > > > > > > which was shared with Trusty, to one more suitable for
> > usage
> > > > > with a
> > > > > > > > > > relational DB like postgresql (but still compatible with
> > big
> > > > > table
> > > > > > > dbs).
> > > > > > > > > > The idea will be to replace generic Storage interface by
> > four
> > > > > > > specific
> > > > > > > > > > interfaces (which will inherit from a common one that
> keeps
> > > the
> > > > > query
> > > > > > > > > part
> > > > > > > > > > at is it. with get and query methods), that will include
> > the
> > > > > required
> > > > > > > > > > modification operations for the four DataIndex "domains":
> > > > > > > > > processinstance,
> > > > > > > > > > usertask, processdefinitions and jobs. Those interfaces
> > will
> > > > > define
> > > > > > > > > methods
> > > > > > > > > > like addNode, addVariable, updateTask, addAttachment.....
> > > that
> > > > > will
> > > > > > > allow
> > > > > > > > > > the persistent layer implementation  to just update the
> > > needed
> > > > > info
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > DB  (for example, for addNode in Postgres, just insert a
> > row
> > > > into
> > > > > > > nodes
> > > > > > > > > > table, for addNode in Mongo, basically the same atomic
> > upsert
> > > > > > > operation
> > > > > > > > > > that is currently done). Therefore, we increase
> performance
> > > for
> > > > > > > Postgres
> > > > > > > > > > and keep the current one for Mongo. The current DB
> schemas
> > > > won't
> > > > > be
> > > > > > > > > > touched.
> > > > > > > > > > Since the code change is large, I do not think I'll be
> able
> > > to
> > > > > have
> > > > > > > the
> > > > > > > > > PR
> > > > > > > > > > ready till next week.
> > > > > > > > > > But before starting, please let me know if that approach
> is
> > > > fine
> > > > > for
> > > > > > > you.
> > > > > > > > > > Best regards.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Nov 24, 2023 at 6:55 PM Alex Porcelli <
> > > > alex@porcelli.me>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thank you Francisco to getting deeper on this…
> > > > > > > > > > >
> > > > > > > > > > > Looking forward to see the results of your suggested
> > > > > improvements.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Nov 24, 2023 at 9:40 AM Francisco Javier Tirado
> > > > Sarti <
> > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I forgot to attach the queries
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Nov 24, 2023 at 3:04 PM Francisco Javier
> Tirado
> > > > > Sarti <
> > > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> Hi,
> > > > > > > > > > > >> A brief update on this topic.
> > > > > > > > > > > >> After doing a simple test with example
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > > > > > ,
> > > > > > > > > > > >> the number of updates over Nodes table is n*n, so we
> > > > manage
> > > > > to
> > > > > > > > > obtain a
> > > > > > > > > > > >> perfect quadratic performance degradation. The
> problem
> > > is
> > > > > worse
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > case
> > > > > > > > > > > >> of Serverless Workflow than in BPMN because we the
> > > number
> > > > of
> > > > > > > nodes
> > > > > > > > > is
> > > > > > > > > > > >> greater than the number of states. In that example N
> > is
> > > > 16,
> > > > > but
> > > > > > > for
> > > > > > > > > a
> > > > > > > > > > > more
> > > > > > > > > > > >> complex workflow it would be certainly large.
> > > > > > > > > > > >> I think that this is more related to how we are
> > handling
> > > > > JPA in
> > > > > > > the
> > > > > > > > > > > code,
> > > > > > > > > > > >> in particular the mapping from model to entity
> > > (basically
> > > > > JPA is
> > > > > > > > > blind
> > > > > > > > > > > and
> > > > > > > > > > > >> has to update all nodes for every write because it
> > > > believes
> > > > > the
> > > > > > > > > node has
> > > > > > > > > > > >> been updated, although it is not) than an issue in
> the
> > > > table
> > > > > > > > > definition.
> > > > > > > > > > > >> In fact, when using JPA, separating the server model
> > > from
> > > > > the
> > > > > > > JPA
> > > > > > > > > > > entity is
> > > > > > > > > > > >> not a good idea, especially if the entity contains
> > > > > collections.
> > > > > > > I
> > > > > > > > > will
> > > > > > > > > > > try
> > > > > > > > > > > >> to change that without breaking anything.
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Wed, Nov 22, 2023 at 12:10 PM Enrique Gonzalez
> > > > Martinez <
> > > > > > > > > > > >> egonzalez@apache.org> wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >>> After the events split you now will need to create
> a
> > > node
> > > > > > > instance
> > > > > > > > > > > >>> model instance of making independent from the
> process
> > > > > instance.
> > > > > > > > > > > >>> That should do the trick.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> Regarding deleting/inserting it was fixed at some
> > > point.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> El mar, 21 nov 2023 a las 20:22, Francisco Javier
> > > Tirado
> > > > > Sarti
> > > > > > > > > > > >>> (<ft...@redhat.com>) escribió:
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > Hi Martin,
> > > > > > > > > > > >>> > I have a task to review performance of
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > ProcessInstanceNodeDataEventMerger
> > > > > > > > > > > >>> > My idea is to reduce the number of delete inserts
> > > when
> > > > > > > processing
> > > > > > > > > > > >>> events
> > > > > > > > > > > >>> > and try to do it incremental.
> > > > > > > > > > > >>> > That should improve performance.
> > > > > > > > > > > >>> > PS:
> > > > > > > > > > > >>> > I was planning to send an e-mail tomorrow
> > announcing
> > > > > that in
> > > > > > > > > case you
> > > > > > > > > > > >>> were
> > > > > > > > > > > >>> > already working on a fix for that. I assume you
> are
> > > not
> > > > > and I
> > > > > > > > > would
> > > > > > > > > > > be
> > > > > > > > > > > >>> > sending a PR soon.
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > On Tue, Nov 21, 2023 at 6:09 PM Martin Weiler
> > > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > wrote:
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > > I looked into the new examples using data-index
> > > > > persistence
> > > > > > > > > addon -
> > > > > > > > > > > >>> Neus'
> > > > > > > > > > > >>> > > PR#1813 [1] for serverless and Pere's branch
> [2]
> > > for
> > > > > > > workflow
> > > > > > > > > > > (great
> > > > > > > > > > > >>> job
> > > > > > > > > > > >>> > > both!) - and they work without issues using
> > single
> > > > > > > requests.
> > > > > > > > > > > >>> However, under
> > > > > > > > > > > >>> > > some load (I used 'ab' for testing with a light
> > > > > > > concurrency of
> > > > > > > > > 10
> > > > > > > > > > > >>> parallel
> > > > > > > > > > > >>> > > requests) I ran into the following problems:
> > > > > > > > > > > >>> > >
> > > > > > > > > > > >>> > > (1) Large number of insert/delete calls (eg.
> for
> > > > tables
> > > > > > > such as
> > > > > > > > > > > >>> nodes,
> > > > > > > > > > > >>> > > definitions, etc)
> > > > > > > > > > > >>> > >
> > > > > > > > > > > >>> > > (2) Hibernate OptimisticLockExceptions /
> > > > > > > StaleStateExceptions
> > > > > > > > > > > >>> > >
> > > > > > > > > > > >>> > > (3) DB deadlocks
> > > > > > > > > > > >>> > >
> > > > > > > > > > > >>> > > (4) Error responses, slow response times
> > > > > > > > > > > >>> > >
> > > > > > > > > > > >>> > > The reason I am reaching out with this topic
> here
> > > is
> > > > to
> > > > > > > find
> > > > > > > > > out if
> > > > > > > > > > > >>> we are
> > > > > > > > > > > >>> > > aware of this issue, and if someone is already
> > > > looking
> > > > > > > into or
> > > > > > > > > > > being
> > > > > > > > > > > >>> > > assigned to it?
> > > > > > > > > > > >>> > >
> > > > > > > > > > > >>> > > Thanks,
> > > > > > > > > > > >>> > > Martin
> > > > > > > > > > > >>> > >
> > > > > > > > > > > >>> > > [1]
> > > > > > > > > > > >>>
> > > > > > >
> > https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > > > > > > > >>> > > [2]
> > > > > > > > > > > >>> > >
> > > > > > > > > > > >>>
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > > > > > > > >>> > >
> > > > > > > > > > > >>> > >
> > > > > > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > > > > >>> > > To unsubscribe, e-mail:
> > > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > >>> > > For additional commands, e-mail:
> > > > > dev-help@kie.apache.org
> > > > > > > > > > > >>> > >
> > > > > > > > > > > >>> > >
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > > > >>> To unsubscribe, e-mail:
> > dev-unsubscribe@kie.apache.org
> > > > > > > > > > > >>> For additional commands, e-mail:
> > > dev-help@kie.apache.org
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>
> > > > > > > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > > > > > To unsubscribe, e-mail:
> dev-unsubscribe@kie.apache.org
> > > > > > > > > > > > For additional commands, e-mail:
> > dev-help@kie.apache.org
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > >
> > > > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > >
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > >
> > > >
> > >
> >
>

RE: [DISCUSSION] Performance issues with data-index persistence addon

[Martin Weiler <mw...@ibm.com.INVALID>]

I changed the test to use MongoDB [1] and I don't see a performance degradation with this setup [2].

Please keep us posted of your findings. Thanks!

Martin

[1] https://github.com/martinweiler/job-service-refactor-test/tree/mongodb
[2] https://drive.google.com/file/d/1NfacXaxJlgRMw4OQ5S20cvkzvaUKUVFj/view?usp=sharing

________________________________________
From: Francisco Javier Tirado Sarti <ft...@redhat.com>
Sent: Wednesday, February 7, 2024 11:40 AM
To: dev@kie.apache.org
Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with data-index persistence addon

yes, it can be index degradation because of size, but I believe (I might be
wrong) the db is too small (yet) for that.
But, eventually, Postgres, when the DB is huge enough, unavoidably will
behave like the graphic that Martin sent.
Since I believe we are not huge enough (yet), lets rule out another issue
by analysing the sql logs (I requested those to Martin offline and he is
going to kindly collect them).
Also Im curious to know if Mongo behave in the same way.

On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez Martinez <
egonzalez@apache.org> wrote:

> Hi Francisco,
> I would highly recommend to check indexes and how the updates work in data
> index to avoid full scan table and lock the full table. Some db are very
> sensitive to that.
>
> El mié, 7 feb 2024, 18:41, Francisco Javier Tirado Sarti <
> ftirados@redhat.com> escribió:
>
> > Hi Martin,
> > While I analyze the data, let me ask you if it is possible to perform
> > another check (similar in a way to disabling data-index like you do) Can
> > you switch to MongoDB persistence and check if the same degradation that
> is
> > there for postgres remains?
> > I do not know if this is feasible but will certainly indicate the problem
> > is on the postgres storage layer and I do not have a clear prediction of
> > what we will see when doing this switch.
> >
> > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler <mw...@ibm.com.invalid>
> > wrote:
> >
> > > Hi Francisco,
> > >
> > > thanks for your work on this important topic!
> > >
> > > I would like to share some test results here, which might help to
> improve
> > > the codebase even further. I am using the jmeter based test case from
> > Pere
> > > and Enrique (thanks guys!) [1] which uses a load of 30 threads to
> > >
> > > 1) start a new process instance (POST)
> > > 2) retrieve tasks for a user (GET)
> > > 3) fetches task details (GET)
> > > 4) complete a task (POST)
> > > 5) execute a query on data-audit
> > >
> > > With this test setup, I noticed that the performance for the POST
> > > requests, in particular the one to start a new process instance,
> degrades
> > > over time - see graph [2]. If I run the same test without data-index,
> > then
> > > there is no such performance degradation [3]. You can find a thread
> dump
> > > captured a few minutes into the first test here [4] that might help to
> > see
> > > some of the contention points.
> > >
> > > I'd appreciate if you could take a look and see if there is something
> > that
> > > can be further improved based on your previous work. If you need any
> > > additional data, let me know, but otherwise it is straightforward to
> run
> > > the jmeter test as well.
> > >
> > > Thanks,
> > > Martin
> > >
> > > [1] https://github.com/pefernan/job-service-refactor-test/
> > > [2]
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > [3]
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > [4]
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > >
> > > ________________________________________
> > > From: Francisco Javier Tirado Sarti <ft...@redhat.com>
> > > Sent: Wednesday, January 17, 2024 9:13 AM
> > > To: dev@kie.apache.org
> > > Cc: Pere Fernandez Perez
> > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with data-index
> > > persistence addon
> > >
> > > Hi Alex,
> > > I did not take times (which depends on a number of variables that
> > > drastically change between environments), but verify that the number of
> > > updates has been reduced drastically without losing functionality,
> which
> > is
> > > objectively a good thing. If before the change, for every node
> executed,
> > we
> > > have an update for every node previously executed, so if a process have
> > 50
> > > nodes to execute, we were performing nearly 50*51/2 updates, which
> gives
> > us
> > > a total of  1275 updates, now we have just one for every node being
> > > executed, implying a total of 50 updates.
> > >
> > >
> > > On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli <al...@porcelli.me>
> wrote:
> > >
> > > > Francisco,
> > > >
> > > > I noticed that your PR has been merged, but I was expecting (at least
> > > > was my understanding from this thread) that before merging some
> > > > benchmark data would be shared in advance - to assess the
> cost/benefit
> > > > of such a decent size change.
> > > >
> > > > Do you have any information to share?
> > > >
> > > > On Sat, Dec 23, 2023 at 4:02 AM Francisco Javier Tirado Sarti
> > > > <ft...@redhat.com> wrote:
> > > > >
> > > > > Yes, as intended, now we have one select and one insert/update per
> > node
> > > > > event.
> > > > > I moved the PR as ready for review and give @Pere Fernandez Perez
> > > > > <pe...@redhat.com> permission to the branch so he can edit it
> in
> > > the
> > > > > next two weeks (Ill be on PTO)  if desired, before merging.
> > > > >
> > > > >
> > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex Porcelli <al...@porcelli.me>
> > > wrote:
> > > > >
> > > > > > Cool, thank you Francisco!
> > > > > >
> > > > > > Did you manage to get some preliminary data about improvements?
> > > > > >
> > > > > > On Thu, Dec 21, 2023 at 11:52 AM Francisco Javier Tirado Sarti
> > > > > > <ft...@redhat.com> wrote:
> > > > > > >
> > > > > > > Yes, after some delay because of quarkus 3 migration. Im
> refining
> > > > this
> > > > > > > draft PR
> > > > > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > >
> > > > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex Porcelli <
> alex@porcelli.me>
> > > > wrote:
> > > > > > >
> > > > > > > > Any update or new findings on this topic?
> > > > > > > >
> > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM Francisco Javier Tirado Sarti
> > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > >
> > > > > > > > > Hi Alex,
> > > > > > > > > After considering different options to improve performance,
> > we
> > > > feel
> > > > > > that
> > > > > > > > it
> > > > > > > > > is time to "partially" move away from the current Map style
> > > > > > interface (
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > > > )
> > > > > > > > > which was shared with Trusty, to one more suitable for
> usage
> > > > with a
> > > > > > > > > relational DB like postgresql (but still compatible with
> big
> > > > table
> > > > > > dbs).
> > > > > > > > > The idea will be to replace generic Storage interface by
> four
> > > > > > specific
> > > > > > > > > interfaces (which will inherit from a common one that keeps
> > the
> > > > query
> > > > > > > > part
> > > > > > > > > at is it. with get and query methods), that will include
> the
> > > > required
> > > > > > > > > modification operations for the four DataIndex "domains":
> > > > > > > > processinstance,
> > > > > > > > > usertask, processdefinitions and jobs. Those interfaces
> will
> > > > define
> > > > > > > > methods
> > > > > > > > > like addNode, addVariable, updateTask, addAttachment.....
> > that
> > > > will
> > > > > > allow
> > > > > > > > > the persistent layer implementation  to just update the
> > needed
> > > > info
> > > > > > in
> > > > > > > > the
> > > > > > > > > DB  (for example, for addNode in Postgres, just insert a
> row
> > > into
> > > > > > nodes
> > > > > > > > > table, for addNode in Mongo, basically the same atomic
> upsert
> > > > > > operation
> > > > > > > > > that is currently done). Therefore, we increase performance
> > for
> > > > > > Postgres
> > > > > > > > > and keep the current one for Mongo. The current DB schemas
> > > won't
> > > > be
> > > > > > > > > touched.
> > > > > > > > > Since the code change is large, I do not think I'll be able
> > to
> > > > have
> > > > > > the
> > > > > > > > PR
> > > > > > > > > ready till next week.
> > > > > > > > > But before starting, please let me know if that approach is
> > > fine
> > > > for
> > > > > > you.
> > > > > > > > > Best regards.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Nov 24, 2023 at 6:55 PM Alex Porcelli <
> > > alex@porcelli.me>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thank you Francisco to getting deeper on this…
> > > > > > > > > >
> > > > > > > > > > Looking forward to see the results of your suggested
> > > > improvements.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Nov 24, 2023 at 9:40 AM Francisco Javier Tirado
> > > Sarti <
> > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > I forgot to attach the queries
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Nov 24, 2023 at 3:04 PM Francisco Javier Tirado
> > > > Sarti <
> > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Hi,
> > > > > > > > > > >> A brief update on this topic.
> > > > > > > > > > >> After doing a simple test with example
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > > > > ,
> > > > > > > > > > >> the number of updates over Nodes table is n*n, so we
> > > manage
> > > > to
> > > > > > > > obtain a
> > > > > > > > > > >> perfect quadratic performance degradation. The problem
> > is
> > > > worse
> > > > > > in
> > > > > > > > the
> > > > > > > > > > case
> > > > > > > > > > >> of Serverless Workflow than in BPMN because we the
> > number
> > > of
> > > > > > nodes
> > > > > > > > is
> > > > > > > > > > >> greater than the number of states. In that example N
> is
> > > 16,
> > > > but
> > > > > > for
> > > > > > > > a
> > > > > > > > > > more
> > > > > > > > > > >> complex workflow it would be certainly large.
> > > > > > > > > > >> I think that this is more related to how we are
> handling
> > > > JPA in
> > > > > > the
> > > > > > > > > > code,
> > > > > > > > > > >> in particular the mapping from model to entity
> > (basically
> > > > JPA is
> > > > > > > > blind
> > > > > > > > > > and
> > > > > > > > > > >> has to update all nodes for every write because it
> > > believes
> > > > the
> > > > > > > > node has
> > > > > > > > > > >> been updated, although it is not) than an issue in the
> > > table
> > > > > > > > definition.
> > > > > > > > > > >> In fact, when using JPA, separating the server model
> > from
> > > > the
> > > > > > JPA
> > > > > > > > > > entity is
> > > > > > > > > > >> not a good idea, especially if the entity contains
> > > > collections.
> > > > > > I
> > > > > > > > will
> > > > > > > > > > try
> > > > > > > > > > >> to change that without breaking anything.
> > > > > > > > > > >>
> > > > > > > > > > >> On Wed, Nov 22, 2023 at 12:10 PM Enrique Gonzalez
> > > Martinez <
> > > > > > > > > > >> egonzalez@apache.org> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >>> After the events split you now will need to create a
> > node
> > > > > > instance
> > > > > > > > > > >>> model instance of making independent from the process
> > > > instance.
> > > > > > > > > > >>> That should do the trick.
> > > > > > > > > > >>>
> > > > > > > > > > >>> Regarding deleting/inserting it was fixed at some
> > point.
> > > > > > > > > > >>>
> > > > > > > > > > >>> El mar, 21 nov 2023 a las 20:22, Francisco Javier
> > Tirado
> > > > Sarti
> > > > > > > > > > >>> (<ft...@redhat.com>) escribió:
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > Hi Martin,
> > > > > > > > > > >>> > I have a task to review performance of
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > ProcessInstanceNodeDataEventMerger
> > > > > > > > > > >>> > My idea is to reduce the number of delete inserts
> > when
> > > > > > processing
> > > > > > > > > > >>> events
> > > > > > > > > > >>> > and try to do it incremental.
> > > > > > > > > > >>> > That should improve performance.
> > > > > > > > > > >>> > PS:
> > > > > > > > > > >>> > I was planning to send an e-mail tomorrow
> announcing
> > > > that in
> > > > > > > > case you
> > > > > > > > > > >>> were
> > > > > > > > > > >>> > already working on a fix for that. I assume you are
> > not
> > > > and I
> > > > > > > > would
> > > > > > > > > > be
> > > > > > > > > > >>> > sending a PR soon.
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > On Tue, Nov 21, 2023 at 6:09 PM Martin Weiler
> > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > wrote:
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > > I looked into the new examples using data-index
> > > > persistence
> > > > > > > > addon -
> > > > > > > > > > >>> Neus'
> > > > > > > > > > >>> > > PR#1813 [1] for serverless and Pere's branch [2]
> > for
> > > > > > workflow
> > > > > > > > > > (great
> > > > > > > > > > >>> job
> > > > > > > > > > >>> > > both!) - and they work without issues using
> single
> > > > > > requests.
> > > > > > > > > > >>> However, under
> > > > > > > > > > >>> > > some load (I used 'ab' for testing with a light
> > > > > > concurrency of
> > > > > > > > 10
> > > > > > > > > > >>> parallel
> > > > > > > > > > >>> > > requests) I ran into the following problems:
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > (1) Large number of insert/delete calls (eg. for
> > > tables
> > > > > > such as
> > > > > > > > > > >>> nodes,
> > > > > > > > > > >>> > > definitions, etc)
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > (2) Hibernate OptimisticLockExceptions /
> > > > > > StaleStateExceptions
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > (3) DB deadlocks
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > (4) Error responses, slow response times
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > The reason I am reaching out with this topic here
> > is
> > > to
> > > > > > find
> > > > > > > > out if
> > > > > > > > > > >>> we are
> > > > > > > > > > >>> > > aware of this issue, and if someone is already
> > > looking
> > > > > > into or
> > > > > > > > > > being
> > > > > > > > > > >>> > > assigned to it?
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > Thanks,
> > > > > > > > > > >>> > > Martin
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > [1]
> > > > > > > > > > >>>
> > > > > >
> https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > > > > > > >>> > > [2]
> > > > > > > > > > >>> > >
> > > > > > > > > > >>>
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
> >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > >
> > > > > > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > > > >>> > > To unsubscribe, e-mail:
> > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > >>> > > For additional commands, e-mail:
> > > > dev-help@kie.apache.org
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > >
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > > > >>> To unsubscribe, e-mail:
> dev-unsubscribe@kie.apache.org
> > > > > > > > > > >>> For additional commands, e-mail:
> > dev-help@kie.apache.org
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > > > > For additional commands, e-mail:
> dev-help@kie.apache.org
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > >
> > > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > For additional commands, e-mail: dev-help@kie.apache.org
> > >
> > >
> >
>

RE: [DISCUSSION] Performance issues with data-index persistence addon

[Martin Weiler <mw...@ibm.com.INVALID>]

I ran another test with hibernate logging enabled, you can find a zip file with all collected data here [1]. Some noteworthy things:

1. I disabled data audit for the following reasons
   #1 not to get flooded with hibernate messages for the *log tables
   #2 to verify it's role in the performance degradation, which is still visible even without data audit
2. The thread dumps show some threads waiting in QuarkusConnectionProvider.getConnection(..), even though I increased the pool size
3. The number of inserts/updates to the nodes and processes table appears to be pretty high
4. The processes and nodes table have no other index besides the PK
5. Database size was around 200MB at the end of the ~20min test, so DB size shouldn't be the issue

Martin

[1] https://drive.google.com/file/d/1PbCRwq0jSeN-sU-nByfcJzEJ7EVGT3tv/view?usp=sharing

________________________________________
From: Francisco Javier Tirado Sarti <ft...@redhat.com>
Sent: Wednesday, February 7, 2024 11:40 AM
To: dev@kie.apache.org
Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with data-index persistence addon

yes, it can be index degradation because of size, but I believe (I might be
wrong) the db is too small (yet) for that.
But, eventually, Postgres, when the DB is huge enough, unavoidably will
behave like the graphic that Martin sent.
Since I believe we are not huge enough (yet), lets rule out another issue
by analysing the sql logs (I requested those to Martin offline and he is
going to kindly collect them).
Also Im curious to know if Mongo behave in the same way.

On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez Martinez <
egonzalez@apache.org> wrote:

> Hi Francisco,
> I would highly recommend to check indexes and how the updates work in data
> index to avoid full scan table and lock the full table. Some db are very
> sensitive to that.
>
> El mié, 7 feb 2024, 18:41, Francisco Javier Tirado Sarti <
> ftirados@redhat.com> escribió:
>
> > Hi Martin,
> > While I analyze the data, let me ask you if it is possible to perform
> > another check (similar in a way to disabling data-index like you do) Can
> > you switch to MongoDB persistence and check if the same degradation that
> is
> > there for postgres remains?
> > I do not know if this is feasible but will certainly indicate the problem
> > is on the postgres storage layer and I do not have a clear prediction of
> > what we will see when doing this switch.
> >
> > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler <mw...@ibm.com.invalid>
> > wrote:
> >
> > > Hi Francisco,
> > >
> > > thanks for your work on this important topic!
> > >
> > > I would like to share some test results here, which might help to
> improve
> > > the codebase even further. I am using the jmeter based test case from
> > Pere
> > > and Enrique (thanks guys!) [1] which uses a load of 30 threads to
> > >
> > > 1) start a new process instance (POST)
> > > 2) retrieve tasks for a user (GET)
> > > 3) fetches task details (GET)
> > > 4) complete a task (POST)
> > > 5) execute a query on data-audit
> > >
> > > With this test setup, I noticed that the performance for the POST
> > > requests, in particular the one to start a new process instance,
> degrades
> > > over time - see graph [2]. If I run the same test without data-index,
> > then
> > > there is no such performance degradation [3]. You can find a thread
> dump
> > > captured a few minutes into the first test here [4] that might help to
> > see
> > > some of the contention points.
> > >
> > > I'd appreciate if you could take a look and see if there is something
> > that
> > > can be further improved based on your previous work. If you need any
> > > additional data, let me know, but otherwise it is straightforward to
> run
> > > the jmeter test as well.
> > >
> > > Thanks,
> > > Martin
> > >
> > > [1] https://github.com/pefernan/job-service-refactor-test/
> > > [2]
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > [3]
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > [4]
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > >
> > > ________________________________________
> > > From: Francisco Javier Tirado Sarti <ft...@redhat.com>
> > > Sent: Wednesday, January 17, 2024 9:13 AM
> > > To: dev@kie.apache.org
> > > Cc: Pere Fernandez Perez
> > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with data-index
> > > persistence addon
> > >
> > > Hi Alex,
> > > I did not take times (which depends on a number of variables that
> > > drastically change between environments), but verify that the number of
> > > updates has been reduced drastically without losing functionality,
> which
> > is
> > > objectively a good thing. If before the change, for every node
> executed,
> > we
> > > have an update for every node previously executed, so if a process have
> > 50
> > > nodes to execute, we were performing nearly 50*51/2 updates, which
> gives
> > us
> > > a total of  1275 updates, now we have just one for every node being
> > > executed, implying a total of 50 updates.
> > >
> > >
> > > On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli <al...@porcelli.me>
> wrote:
> > >
> > > > Francisco,
> > > >
> > > > I noticed that your PR has been merged, but I was expecting (at least
> > > > was my understanding from this thread) that before merging some
> > > > benchmark data would be shared in advance - to assess the
> cost/benefit
> > > > of such a decent size change.
> > > >
> > > > Do you have any information to share?
> > > >
> > > > On Sat, Dec 23, 2023 at 4:02 AM Francisco Javier Tirado Sarti
> > > > <ft...@redhat.com> wrote:
> > > > >
> > > > > Yes, as intended, now we have one select and one insert/update per
> > node
> > > > > event.
> > > > > I moved the PR as ready for review and give @Pere Fernandez Perez
> > > > > <pe...@redhat.com> permission to the branch so he can edit it
> in
> > > the
> > > > > next two weeks (Ill be on PTO)  if desired, before merging.
> > > > >
> > > > >
> > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex Porcelli <al...@porcelli.me>
> > > wrote:
> > > > >
> > > > > > Cool, thank you Francisco!
> > > > > >
> > > > > > Did you manage to get some preliminary data about improvements?
> > > > > >
> > > > > > On Thu, Dec 21, 2023 at 11:52 AM Francisco Javier Tirado Sarti
> > > > > > <ft...@redhat.com> wrote:
> > > > > > >
> > > > > > > Yes, after some delay because of quarkus 3 migration. Im
> refining
> > > > this
> > > > > > > draft PR
> > > > > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > >
> > > > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex Porcelli <
> alex@porcelli.me>
> > > > wrote:
> > > > > > >
> > > > > > > > Any update or new findings on this topic?
> > > > > > > >
> > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM Francisco Javier Tirado Sarti
> > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > >
> > > > > > > > > Hi Alex,
> > > > > > > > > After considering different options to improve performance,
> > we
> > > > feel
> > > > > > that
> > > > > > > > it
> > > > > > > > > is time to "partially" move away from the current Map style
> > > > > > interface (
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > > > )
> > > > > > > > > which was shared with Trusty, to one more suitable for
> usage
> > > > with a
> > > > > > > > > relational DB like postgresql (but still compatible with
> big
> > > > table
> > > > > > dbs).
> > > > > > > > > The idea will be to replace generic Storage interface by
> four
> > > > > > specific
> > > > > > > > > interfaces (which will inherit from a common one that keeps
> > the
> > > > query
> > > > > > > > part
> > > > > > > > > at is it. with get and query methods), that will include
> the
> > > > required
> > > > > > > > > modification operations for the four DataIndex "domains":
> > > > > > > > processinstance,
> > > > > > > > > usertask, processdefinitions and jobs. Those interfaces
> will
> > > > define
> > > > > > > > methods
> > > > > > > > > like addNode, addVariable, updateTask, addAttachment.....
> > that
> > > > will
> > > > > > allow
> > > > > > > > > the persistent layer implementation  to just update the
> > needed
> > > > info
> > > > > > in
> > > > > > > > the
> > > > > > > > > DB  (for example, for addNode in Postgres, just insert a
> row
> > > into
> > > > > > nodes
> > > > > > > > > table, for addNode in Mongo, basically the same atomic
> upsert
> > > > > > operation
> > > > > > > > > that is currently done). Therefore, we increase performance
> > for
> > > > > > Postgres
> > > > > > > > > and keep the current one for Mongo. The current DB schemas
> > > won't
> > > > be
> > > > > > > > > touched.
> > > > > > > > > Since the code change is large, I do not think I'll be able
> > to
> > > > have
> > > > > > the
> > > > > > > > PR
> > > > > > > > > ready till next week.
> > > > > > > > > But before starting, please let me know if that approach is
> > > fine
> > > > for
> > > > > > you.
> > > > > > > > > Best regards.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Nov 24, 2023 at 6:55 PM Alex Porcelli <
> > > alex@porcelli.me>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thank you Francisco to getting deeper on this…
> > > > > > > > > >
> > > > > > > > > > Looking forward to see the results of your suggested
> > > > improvements.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Nov 24, 2023 at 9:40 AM Francisco Javier Tirado
> > > Sarti <
> > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > I forgot to attach the queries
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Nov 24, 2023 at 3:04 PM Francisco Javier Tirado
> > > > Sarti <
> > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Hi,
> > > > > > > > > > >> A brief update on this topic.
> > > > > > > > > > >> After doing a simple test with example
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > > > > ,
> > > > > > > > > > >> the number of updates over Nodes table is n*n, so we
> > > manage
> > > > to
> > > > > > > > obtain a
> > > > > > > > > > >> perfect quadratic performance degradation. The problem
> > is
> > > > worse
> > > > > > in
> > > > > > > > the
> > > > > > > > > > case
> > > > > > > > > > >> of Serverless Workflow than in BPMN because we the
> > number
> > > of
> > > > > > nodes
> > > > > > > > is
> > > > > > > > > > >> greater than the number of states. In that example N
> is
> > > 16,
> > > > but
> > > > > > for
> > > > > > > > a
> > > > > > > > > > more
> > > > > > > > > > >> complex workflow it would be certainly large.
> > > > > > > > > > >> I think that this is more related to how we are
> handling
> > > > JPA in
> > > > > > the
> > > > > > > > > > code,
> > > > > > > > > > >> in particular the mapping from model to entity
> > (basically
> > > > JPA is
> > > > > > > > blind
> > > > > > > > > > and
> > > > > > > > > > >> has to update all nodes for every write because it
> > > believes
> > > > the
> > > > > > > > node has
> > > > > > > > > > >> been updated, although it is not) than an issue in the
> > > table
> > > > > > > > definition.
> > > > > > > > > > >> In fact, when using JPA, separating the server model
> > from
> > > > the
> > > > > > JPA
> > > > > > > > > > entity is
> > > > > > > > > > >> not a good idea, especially if the entity contains
> > > > collections.
> > > > > > I
> > > > > > > > will
> > > > > > > > > > try
> > > > > > > > > > >> to change that without breaking anything.
> > > > > > > > > > >>
> > > > > > > > > > >> On Wed, Nov 22, 2023 at 12:10 PM Enrique Gonzalez
> > > Martinez <
> > > > > > > > > > >> egonzalez@apache.org> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >>> After the events split you now will need to create a
> > node
> > > > > > instance
> > > > > > > > > > >>> model instance of making independent from the process
> > > > instance.
> > > > > > > > > > >>> That should do the trick.
> > > > > > > > > > >>>
> > > > > > > > > > >>> Regarding deleting/inserting it was fixed at some
> > point.
> > > > > > > > > > >>>
> > > > > > > > > > >>> El mar, 21 nov 2023 a las 20:22, Francisco Javier
> > Tirado
> > > > Sarti
> > > > > > > > > > >>> (<ft...@redhat.com>) escribió:
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > Hi Martin,
> > > > > > > > > > >>> > I have a task to review performance of
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > ProcessInstanceNodeDataEventMerger
> > > > > > > > > > >>> > My idea is to reduce the number of delete inserts
> > when
> > > > > > processing
> > > > > > > > > > >>> events
> > > > > > > > > > >>> > and try to do it incremental.
> > > > > > > > > > >>> > That should improve performance.
> > > > > > > > > > >>> > PS:
> > > > > > > > > > >>> > I was planning to send an e-mail tomorrow
> announcing
> > > > that in
> > > > > > > > case you
> > > > > > > > > > >>> were
> > > > > > > > > > >>> > already working on a fix for that. I assume you are
> > not
> > > > and I
> > > > > > > > would
> > > > > > > > > > be
> > > > > > > > > > >>> > sending a PR soon.
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > On Tue, Nov 21, 2023 at 6:09 PM Martin Weiler
> > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > wrote:
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > > I looked into the new examples using data-index
> > > > persistence
> > > > > > > > addon -
> > > > > > > > > > >>> Neus'
> > > > > > > > > > >>> > > PR#1813 [1] for serverless and Pere's branch [2]
> > for
> > > > > > workflow
> > > > > > > > > > (great
> > > > > > > > > > >>> job
> > > > > > > > > > >>> > > both!) - and they work without issues using
> single
> > > > > > requests.
> > > > > > > > > > >>> However, under
> > > > > > > > > > >>> > > some load (I used 'ab' for testing with a light
> > > > > > concurrency of
> > > > > > > > 10
> > > > > > > > > > >>> parallel
> > > > > > > > > > >>> > > requests) I ran into the following problems:
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > (1) Large number of insert/delete calls (eg. for
> > > tables
> > > > > > such as
> > > > > > > > > > >>> nodes,
> > > > > > > > > > >>> > > definitions, etc)
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > (2) Hibernate OptimisticLockExceptions /
> > > > > > StaleStateExceptions
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > (3) DB deadlocks
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > (4) Error responses, slow response times
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > The reason I am reaching out with this topic here
> > is
> > > to
> > > > > > find
> > > > > > > > out if
> > > > > > > > > > >>> we are
> > > > > > > > > > >>> > > aware of this issue, and if someone is already
> > > looking
> > > > > > into or
> > > > > > > > > > being
> > > > > > > > > > >>> > > assigned to it?
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > Thanks,
> > > > > > > > > > >>> > > Martin
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > [1]
> > > > > > > > > > >>>
> > > > > >
> https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > > > > > > >>> > > [2]
> > > > > > > > > > >>> > >
> > > > > > > > > > >>>
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
> >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > >
> > > > > > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > > > >>> > > To unsubscribe, e-mail:
> > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > >>> > > For additional commands, e-mail:
> > > > dev-help@kie.apache.org
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > >
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > > > >>> To unsubscribe, e-mail:
> dev-unsubscribe@kie.apache.org
> > > > > > > > > > >>> For additional commands, e-mail:
> > dev-help@kie.apache.org
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > > > > For additional commands, e-mail:
> dev-help@kie.apache.org
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > >
> > > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > For additional commands, e-mail: dev-help@kie.apache.org
> > >
> > >
> >
>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Francisco Javier Tirado Sarti <ft...@redhat.com>]

yes, it can be index degradation because of size, but I believe (I might be
wrong) the db is too small (yet) for that.
But, eventually, Postgres, when the DB is huge enough, unavoidably will
behave like the graphic that Martin sent.
Since I believe we are not huge enough (yet), lets rule out another issue
by analysing the sql logs (I requested those to Martin offline and he is
going to kindly collect them).
Also Im curious to know if Mongo behave in the same way.

On Wed, Feb 7, 2024 at 7:25 PM Enrique Gonzalez Martinez <
egonzalez@apache.org> wrote:

> Hi Francisco,
> I would highly recommend to check indexes and how the updates work in data
> index to avoid full scan table and lock the full table. Some db are very
> sensitive to that.
>
> El mié, 7 feb 2024, 18:41, Francisco Javier Tirado Sarti <
> ftirados@redhat.com> escribió:
>
> > Hi Martin,
> > While I analyze the data, let me ask you if it is possible to perform
> > another check (similar in a way to disabling data-index like you do) Can
> > you switch to MongoDB persistence and check if the same degradation that
> is
> > there for postgres remains?
> > I do not know if this is feasible but will certainly indicate the problem
> > is on the postgres storage layer and I do not have a clear prediction of
> > what we will see when doing this switch.
> >
> > On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler <mw...@ibm.com.invalid>
> > wrote:
> >
> > > Hi Francisco,
> > >
> > > thanks for your work on this important topic!
> > >
> > > I would like to share some test results here, which might help to
> improve
> > > the codebase even further. I am using the jmeter based test case from
> > Pere
> > > and Enrique (thanks guys!) [1] which uses a load of 30 threads to
> > >
> > > 1) start a new process instance (POST)
> > > 2) retrieve tasks for a user (GET)
> > > 3) fetches task details (GET)
> > > 4) complete a task (POST)
> > > 5) execute a query on data-audit
> > >
> > > With this test setup, I noticed that the performance for the POST
> > > requests, in particular the one to start a new process instance,
> degrades
> > > over time - see graph [2]. If I run the same test without data-index,
> > then
> > > there is no such performance degradation [3]. You can find a thread
> dump
> > > captured a few minutes into the first test here [4] that might help to
> > see
> > > some of the contention points.
> > >
> > > I'd appreciate if you could take a look and see if there is something
> > that
> > > can be further improved based on your previous work. If you need any
> > > additional data, let me know, but otherwise it is straightforward to
> run
> > > the jmeter test as well.
> > >
> > > Thanks,
> > > Martin
> > >
> > > [1] https://github.com/pefernan/job-service-refactor-test/
> > > [2]
> > >
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > > [3]
> > >
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > > [4]
> > >
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> > >
> > > ________________________________________
> > > From: Francisco Javier Tirado Sarti <ft...@redhat.com>
> > > Sent: Wednesday, January 17, 2024 9:13 AM
> > > To: dev@kie.apache.org
> > > Cc: Pere Fernandez Perez
> > > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with data-index
> > > persistence addon
> > >
> > > Hi Alex,
> > > I did not take times (which depends on a number of variables that
> > > drastically change between environments), but verify that the number of
> > > updates has been reduced drastically without losing functionality,
> which
> > is
> > > objectively a good thing. If before the change, for every node
> executed,
> > we
> > > have an update for every node previously executed, so if a process have
> > 50
> > > nodes to execute, we were performing nearly 50*51/2 updates, which
> gives
> > us
> > > a total of  1275 updates, now we have just one for every node being
> > > executed, implying a total of 50 updates.
> > >
> > >
> > > On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli <al...@porcelli.me>
> wrote:
> > >
> > > > Francisco,
> > > >
> > > > I noticed that your PR has been merged, but I was expecting (at least
> > > > was my understanding from this thread) that before merging some
> > > > benchmark data would be shared in advance - to assess the
> cost/benefit
> > > > of such a decent size change.
> > > >
> > > > Do you have any information to share?
> > > >
> > > > On Sat, Dec 23, 2023 at 4:02 AM Francisco Javier Tirado Sarti
> > > > <ft...@redhat.com> wrote:
> > > > >
> > > > > Yes, as intended, now we have one select and one insert/update per
> > node
> > > > > event.
> > > > > I moved the PR as ready for review and give @Pere Fernandez Perez
> > > > > <pe...@redhat.com> permission to the branch so he can edit it
> in
> > > the
> > > > > next two weeks (Ill be on PTO)  if desired, before merging.
> > > > >
> > > > >
> > > > > On Thu, Dec 21, 2023 at 5:58 PM Alex Porcelli <al...@porcelli.me>
> > > wrote:
> > > > >
> > > > > > Cool, thank you Francisco!
> > > > > >
> > > > > > Did you manage to get some preliminary data about improvements?
> > > > > >
> > > > > > On Thu, Dec 21, 2023 at 11:52 AM Francisco Javier Tirado Sarti
> > > > > > <ft...@redhat.com> wrote:
> > > > > > >
> > > > > > > Yes, after some delay because of quarkus 3 migration. Im
> refining
> > > > this
> > > > > > > draft PR
> > > > > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > > >
> > > > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex Porcelli <
> alex@porcelli.me>
> > > > wrote:
> > > > > > >
> > > > > > > > Any update or new findings on this topic?
> > > > > > > >
> > > > > > > > On Tue, Nov 28, 2023 at 8:38 AM Francisco Javier Tirado Sarti
> > > > > > > > <ft...@redhat.com> wrote:
> > > > > > > > >
> > > > > > > > > Hi Alex,
> > > > > > > > > After considering different options to improve performance,
> > we
> > > > feel
> > > > > > that
> > > > > > > > it
> > > > > > > > > is time to "partially" move away from the current Map style
> > > > > > interface (
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > > > )
> > > > > > > > > which was shared with Trusty, to one more suitable for
> usage
> > > > with a
> > > > > > > > > relational DB like postgresql (but still compatible with
> big
> > > > table
> > > > > > dbs).
> > > > > > > > > The idea will be to replace generic Storage interface by
> four
> > > > > > specific
> > > > > > > > > interfaces (which will inherit from a common one that keeps
> > the
> > > > query
> > > > > > > > part
> > > > > > > > > at is it. with get and query methods), that will include
> the
> > > > required
> > > > > > > > > modification operations for the four DataIndex "domains":
> > > > > > > > processinstance,
> > > > > > > > > usertask, processdefinitions and jobs. Those interfaces
> will
> > > > define
> > > > > > > > methods
> > > > > > > > > like addNode, addVariable, updateTask, addAttachment.....
> > that
> > > > will
> > > > > > allow
> > > > > > > > > the persistent layer implementation  to just update the
> > needed
> > > > info
> > > > > > in
> > > > > > > > the
> > > > > > > > > DB  (for example, for addNode in Postgres, just insert a
> row
> > > into
> > > > > > nodes
> > > > > > > > > table, for addNode in Mongo, basically the same atomic
> upsert
> > > > > > operation
> > > > > > > > > that is currently done). Therefore, we increase performance
> > for
> > > > > > Postgres
> > > > > > > > > and keep the current one for Mongo. The current DB schemas
> > > won't
> > > > be
> > > > > > > > > touched.
> > > > > > > > > Since the code change is large, I do not think I'll be able
> > to
> > > > have
> > > > > > the
> > > > > > > > PR
> > > > > > > > > ready till next week.
> > > > > > > > > But before starting, please let me know if that approach is
> > > fine
> > > > for
> > > > > > you.
> > > > > > > > > Best regards.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Nov 24, 2023 at 6:55 PM Alex Porcelli <
> > > alex@porcelli.me>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thank you Francisco to getting deeper on this…
> > > > > > > > > >
> > > > > > > > > > Looking forward to see the results of your suggested
> > > > improvements.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Nov 24, 2023 at 9:40 AM Francisco Javier Tirado
> > > Sarti <
> > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > I forgot to attach the queries
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Nov 24, 2023 at 3:04 PM Francisco Javier Tirado
> > > > Sarti <
> > > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Hi,
> > > > > > > > > > >> A brief update on this topic.
> > > > > > > > > > >> After doing a simple test with example
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > > > > ,
> > > > > > > > > > >> the number of updates over Nodes table is n*n, so we
> > > manage
> > > > to
> > > > > > > > obtain a
> > > > > > > > > > >> perfect quadratic performance degradation. The problem
> > is
> > > > worse
> > > > > > in
> > > > > > > > the
> > > > > > > > > > case
> > > > > > > > > > >> of Serverless Workflow than in BPMN because we the
> > number
> > > of
> > > > > > nodes
> > > > > > > > is
> > > > > > > > > > >> greater than the number of states. In that example N
> is
> > > 16,
> > > > but
> > > > > > for
> > > > > > > > a
> > > > > > > > > > more
> > > > > > > > > > >> complex workflow it would be certainly large.
> > > > > > > > > > >> I think that this is more related to how we are
> handling
> > > > JPA in
> > > > > > the
> > > > > > > > > > code,
> > > > > > > > > > >> in particular the mapping from model to entity
> > (basically
> > > > JPA is
> > > > > > > > blind
> > > > > > > > > > and
> > > > > > > > > > >> has to update all nodes for every write because it
> > > believes
> > > > the
> > > > > > > > node has
> > > > > > > > > > >> been updated, although it is not) than an issue in the
> > > table
> > > > > > > > definition.
> > > > > > > > > > >> In fact, when using JPA, separating the server model
> > from
> > > > the
> > > > > > JPA
> > > > > > > > > > entity is
> > > > > > > > > > >> not a good idea, especially if the entity contains
> > > > collections.
> > > > > > I
> > > > > > > > will
> > > > > > > > > > try
> > > > > > > > > > >> to change that without breaking anything.
> > > > > > > > > > >>
> > > > > > > > > > >> On Wed, Nov 22, 2023 at 12:10 PM Enrique Gonzalez
> > > Martinez <
> > > > > > > > > > >> egonzalez@apache.org> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >>> After the events split you now will need to create a
> > node
> > > > > > instance
> > > > > > > > > > >>> model instance of making independent from the process
> > > > instance.
> > > > > > > > > > >>> That should do the trick.
> > > > > > > > > > >>>
> > > > > > > > > > >>> Regarding deleting/inserting it was fixed at some
> > point.
> > > > > > > > > > >>>
> > > > > > > > > > >>> El mar, 21 nov 2023 a las 20:22, Francisco Javier
> > Tirado
> > > > Sarti
> > > > > > > > > > >>> (<ft...@redhat.com>) escribió:
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > Hi Martin,
> > > > > > > > > > >>> > I have a task to review performance of
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > ProcessInstanceNodeDataEventMerger
> > > > > > > > > > >>> > My idea is to reduce the number of delete inserts
> > when
> > > > > > processing
> > > > > > > > > > >>> events
> > > > > > > > > > >>> > and try to do it incremental.
> > > > > > > > > > >>> > That should improve performance.
> > > > > > > > > > >>> > PS:
> > > > > > > > > > >>> > I was planning to send an e-mail tomorrow
> announcing
> > > > that in
> > > > > > > > case you
> > > > > > > > > > >>> were
> > > > > > > > > > >>> > already working on a fix for that. I assume you are
> > not
> > > > and I
> > > > > > > > would
> > > > > > > > > > be
> > > > > > > > > > >>> > sending a PR soon.
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > On Tue, Nov 21, 2023 at 6:09 PM Martin Weiler
> > > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > wrote:
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > > I looked into the new examples using data-index
> > > > persistence
> > > > > > > > addon -
> > > > > > > > > > >>> Neus'
> > > > > > > > > > >>> > > PR#1813 [1] for serverless and Pere's branch [2]
> > for
> > > > > > workflow
> > > > > > > > > > (great
> > > > > > > > > > >>> job
> > > > > > > > > > >>> > > both!) - and they work without issues using
> single
> > > > > > requests.
> > > > > > > > > > >>> However, under
> > > > > > > > > > >>> > > some load (I used 'ab' for testing with a light
> > > > > > concurrency of
> > > > > > > > 10
> > > > > > > > > > >>> parallel
> > > > > > > > > > >>> > > requests) I ran into the following problems:
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > (1) Large number of insert/delete calls (eg. for
> > > tables
> > > > > > such as
> > > > > > > > > > >>> nodes,
> > > > > > > > > > >>> > > definitions, etc)
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > (2) Hibernate OptimisticLockExceptions /
> > > > > > StaleStateExceptions
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > (3) DB deadlocks
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > (4) Error responses, slow response times
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > The reason I am reaching out with this topic here
> > is
> > > to
> > > > > > find
> > > > > > > > out if
> > > > > > > > > > >>> we are
> > > > > > > > > > >>> > > aware of this issue, and if someone is already
> > > looking
> > > > > > into or
> > > > > > > > > > being
> > > > > > > > > > >>> > > assigned to it?
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > Thanks,
> > > > > > > > > > >>> > > Martin
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > [1]
> > > > > > > > > > >>>
> > > > > >
> https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > > > > > > >>> > > [2]
> > > > > > > > > > >>> > >
> > > > > > > > > > >>>
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
> >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > >
> > > > > > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > > > >>> > > To unsubscribe, e-mail:
> > > dev-unsubscribe@kie.apache.org
> > > > > > > > > > >>> > > For additional commands, e-mail:
> > > > dev-help@kie.apache.org
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > >
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > > > >>> To unsubscribe, e-mail:
> dev-unsubscribe@kie.apache.org
> > > > > > > > > > >>> For additional commands, e-mail:
> > dev-help@kie.apache.org
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > > > > For additional commands, e-mail:
> dev-help@kie.apache.org
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > >
> > > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > For additional commands, e-mail: dev-help@kie.apache.org
> > >
> > >
> >
>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Enrique Gonzalez Martinez <eg...@apache.org>]

Hi Francisco,
I would highly recommend to check indexes and how the updates work in data
index to avoid full scan table and lock the full table. Some db are very
sensitive to that.

El mié, 7 feb 2024, 18:41, Francisco Javier Tirado Sarti <
ftirados@redhat.com> escribió:

> Hi Martin,
> While I analyze the data, let me ask you if it is possible to perform
> another check (similar in a way to disabling data-index like you do) Can
> you switch to MongoDB persistence and check if the same degradation that is
> there for postgres remains?
> I do not know if this is feasible but will certainly indicate the problem
> is on the postgres storage layer and I do not have a clear prediction of
> what we will see when doing this switch.
>
> On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler <mw...@ibm.com.invalid>
> wrote:
>
> > Hi Francisco,
> >
> > thanks for your work on this important topic!
> >
> > I would like to share some test results here, which might help to improve
> > the codebase even further. I am using the jmeter based test case from
> Pere
> > and Enrique (thanks guys!) [1] which uses a load of 30 threads to
> >
> > 1) start a new process instance (POST)
> > 2) retrieve tasks for a user (GET)
> > 3) fetches task details (GET)
> > 4) complete a task (POST)
> > 5) execute a query on data-audit
> >
> > With this test setup, I noticed that the performance for the POST
> > requests, in particular the one to start a new process instance, degrades
> > over time - see graph [2]. If I run the same test without data-index,
> then
> > there is no such performance degradation [3]. You can find a thread dump
> > captured a few minutes into the first test here [4] that might help to
> see
> > some of the contention points.
> >
> > I'd appreciate if you could take a look and see if there is something
> that
> > can be further improved based on your previous work. If you need any
> > additional data, let me know, but otherwise it is straightforward to run
> > the jmeter test as well.
> >
> > Thanks,
> > Martin
> >
> > [1] https://github.com/pefernan/job-service-refactor-test/
> > [2]
> >
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> > [3]
> >
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> > [4]
> >
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
> >
> > ________________________________________
> > From: Francisco Javier Tirado Sarti <ft...@redhat.com>
> > Sent: Wednesday, January 17, 2024 9:13 AM
> > To: dev@kie.apache.org
> > Cc: Pere Fernandez Perez
> > Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with data-index
> > persistence addon
> >
> > Hi Alex,
> > I did not take times (which depends on a number of variables that
> > drastically change between environments), but verify that the number of
> > updates has been reduced drastically without losing functionality, which
> is
> > objectively a good thing. If before the change, for every node executed,
> we
> > have an update for every node previously executed, so if a process have
> 50
> > nodes to execute, we were performing nearly 50*51/2 updates, which gives
> us
> > a total of  1275 updates, now we have just one for every node being
> > executed, implying a total of 50 updates.
> >
> >
> > On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli <al...@porcelli.me> wrote:
> >
> > > Francisco,
> > >
> > > I noticed that your PR has been merged, but I was expecting (at least
> > > was my understanding from this thread) that before merging some
> > > benchmark data would be shared in advance - to assess the cost/benefit
> > > of such a decent size change.
> > >
> > > Do you have any information to share?
> > >
> > > On Sat, Dec 23, 2023 at 4:02 AM Francisco Javier Tirado Sarti
> > > <ft...@redhat.com> wrote:
> > > >
> > > > Yes, as intended, now we have one select and one insert/update per
> node
> > > > event.
> > > > I moved the PR as ready for review and give @Pere Fernandez Perez
> > > > <pe...@redhat.com> permission to the branch so he can edit it  in
> > the
> > > > next two weeks (Ill be on PTO)  if desired, before merging.
> > > >
> > > >
> > > > On Thu, Dec 21, 2023 at 5:58 PM Alex Porcelli <al...@porcelli.me>
> > wrote:
> > > >
> > > > > Cool, thank you Francisco!
> > > > >
> > > > > Did you manage to get some preliminary data about improvements?
> > > > >
> > > > > On Thu, Dec 21, 2023 at 11:52 AM Francisco Javier Tirado Sarti
> > > > > <ft...@redhat.com> wrote:
> > > > > >
> > > > > > Yes, after some delay because of quarkus 3 migration. Im refining
> > > this
> > > > > > draft PR
> > > > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > > >
> > > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex Porcelli <al...@porcelli.me>
> > > wrote:
> > > > > >
> > > > > > > Any update or new findings on this topic?
> > > > > > >
> > > > > > > On Tue, Nov 28, 2023 at 8:38 AM Francisco Javier Tirado Sarti
> > > > > > > <ft...@redhat.com> wrote:
> > > > > > > >
> > > > > > > > Hi Alex,
> > > > > > > > After considering different options to improve performance,
> we
> > > feel
> > > > > that
> > > > > > > it
> > > > > > > > is time to "partially" move away from the current Map style
> > > > > interface (
> > > > > > > >
> > > > > > >
> > > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > > )
> > > > > > > > which was shared with Trusty, to one more suitable for usage
> > > with a
> > > > > > > > relational DB like postgresql (but still compatible with big
> > > table
> > > > > dbs).
> > > > > > > > The idea will be to replace generic Storage interface by four
> > > > > specific
> > > > > > > > interfaces (which will inherit from a common one that keeps
> the
> > > query
> > > > > > > part
> > > > > > > > at is it. with get and query methods), that will include the
> > > required
> > > > > > > > modification operations for the four DataIndex "domains":
> > > > > > > processinstance,
> > > > > > > > usertask, processdefinitions and jobs. Those interfaces will
> > > define
> > > > > > > methods
> > > > > > > > like addNode, addVariable, updateTask, addAttachment.....
> that
> > > will
> > > > > allow
> > > > > > > > the persistent layer implementation  to just update the
> needed
> > > info
> > > > > in
> > > > > > > the
> > > > > > > > DB  (for example, for addNode in Postgres, just insert a row
> > into
> > > > > nodes
> > > > > > > > table, for addNode in Mongo, basically the same atomic upsert
> > > > > operation
> > > > > > > > that is currently done). Therefore, we increase performance
> for
> > > > > Postgres
> > > > > > > > and keep the current one for Mongo. The current DB schemas
> > won't
> > > be
> > > > > > > > touched.
> > > > > > > > Since the code change is large, I do not think I'll be able
> to
> > > have
> > > > > the
> > > > > > > PR
> > > > > > > > ready till next week.
> > > > > > > > But before starting, please let me know if that approach is
> > fine
> > > for
> > > > > you.
> > > > > > > > Best regards.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Nov 24, 2023 at 6:55 PM Alex Porcelli <
> > alex@porcelli.me>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Thank you Francisco to getting deeper on this…
> > > > > > > > >
> > > > > > > > > Looking forward to see the results of your suggested
> > > improvements.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Nov 24, 2023 at 9:40 AM Francisco Javier Tirado
> > Sarti <
> > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > >
> > > > > > > > > > I forgot to attach the queries
> > > > > > > > > >
> > > > > > > > > > On Fri, Nov 24, 2023 at 3:04 PM Francisco Javier Tirado
> > > Sarti <
> > > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi,
> > > > > > > > > >> A brief update on this topic.
> > > > > > > > > >> After doing a simple test with example
> > > > > > > > > >>
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > > > ,
> > > > > > > > > >> the number of updates over Nodes table is n*n, so we
> > manage
> > > to
> > > > > > > obtain a
> > > > > > > > > >> perfect quadratic performance degradation. The problem
> is
> > > worse
> > > > > in
> > > > > > > the
> > > > > > > > > case
> > > > > > > > > >> of Serverless Workflow than in BPMN because we the
> number
> > of
> > > > > nodes
> > > > > > > is
> > > > > > > > > >> greater than the number of states. In that example N is
> > 16,
> > > but
> > > > > for
> > > > > > > a
> > > > > > > > > more
> > > > > > > > > >> complex workflow it would be certainly large.
> > > > > > > > > >> I think that this is more related to how we are handling
> > > JPA in
> > > > > the
> > > > > > > > > code,
> > > > > > > > > >> in particular the mapping from model to entity
> (basically
> > > JPA is
> > > > > > > blind
> > > > > > > > > and
> > > > > > > > > >> has to update all nodes for every write because it
> > believes
> > > the
> > > > > > > node has
> > > > > > > > > >> been updated, although it is not) than an issue in the
> > table
> > > > > > > definition.
> > > > > > > > > >> In fact, when using JPA, separating the server model
> from
> > > the
> > > > > JPA
> > > > > > > > > entity is
> > > > > > > > > >> not a good idea, especially if the entity contains
> > > collections.
> > > > > I
> > > > > > > will
> > > > > > > > > try
> > > > > > > > > >> to change that without breaking anything.
> > > > > > > > > >>
> > > > > > > > > >> On Wed, Nov 22, 2023 at 12:10 PM Enrique Gonzalez
> > Martinez <
> > > > > > > > > >> egonzalez@apache.org> wrote:
> > > > > > > > > >>
> > > > > > > > > >>> After the events split you now will need to create a
> node
> > > > > instance
> > > > > > > > > >>> model instance of making independent from the process
> > > instance.
> > > > > > > > > >>> That should do the trick.
> > > > > > > > > >>>
> > > > > > > > > >>> Regarding deleting/inserting it was fixed at some
> point.
> > > > > > > > > >>>
> > > > > > > > > >>> El mar, 21 nov 2023 a las 20:22, Francisco Javier
> Tirado
> > > Sarti
> > > > > > > > > >>> (<ft...@redhat.com>) escribió:
> > > > > > > > > >>> >
> > > > > > > > > >>> > Hi Martin,
> > > > > > > > > >>> > I have a task to review performance of
> > > > > > > > > >>> >
> > > > > > > > > >>> > ProcessInstanceNodeDataEventMerger
> > > > > > > > > >>> > My idea is to reduce the number of delete inserts
> when
> > > > > processing
> > > > > > > > > >>> events
> > > > > > > > > >>> > and try to do it incremental.
> > > > > > > > > >>> > That should improve performance.
> > > > > > > > > >>> > PS:
> > > > > > > > > >>> > I was planning to send an e-mail tomorrow announcing
> > > that in
> > > > > > > case you
> > > > > > > > > >>> were
> > > > > > > > > >>> > already working on a fix for that. I assume you are
> not
> > > and I
> > > > > > > would
> > > > > > > > > be
> > > > > > > > > >>> > sending a PR soon.
> > > > > > > > > >>> >
> > > > > > > > > >>> > On Tue, Nov 21, 2023 at 6:09 PM Martin Weiler
> > > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > > >>> >
> > > > > > > > > >>> > wrote:
> > > > > > > > > >>> >
> > > > > > > > > >>> > > I looked into the new examples using data-index
> > > persistence
> > > > > > > addon -
> > > > > > > > > >>> Neus'
> > > > > > > > > >>> > > PR#1813 [1] for serverless and Pere's branch [2]
> for
> > > > > workflow
> > > > > > > > > (great
> > > > > > > > > >>> job
> > > > > > > > > >>> > > both!) - and they work without issues using single
> > > > > requests.
> > > > > > > > > >>> However, under
> > > > > > > > > >>> > > some load (I used 'ab' for testing with a light
> > > > > concurrency of
> > > > > > > 10
> > > > > > > > > >>> parallel
> > > > > > > > > >>> > > requests) I ran into the following problems:
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > (1) Large number of insert/delete calls (eg. for
> > tables
> > > > > such as
> > > > > > > > > >>> nodes,
> > > > > > > > > >>> > > definitions, etc)
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > (2) Hibernate OptimisticLockExceptions /
> > > > > StaleStateExceptions
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > (3) DB deadlocks
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > (4) Error responses, slow response times
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > The reason I am reaching out with this topic here
> is
> > to
> > > > > find
> > > > > > > out if
> > > > > > > > > >>> we are
> > > > > > > > > >>> > > aware of this issue, and if someone is already
> > looking
> > > > > into or
> > > > > > > > > being
> > > > > > > > > >>> > > assigned to it?
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > Thanks,
> > > > > > > > > >>> > > Martin
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > [1]
> > > > > > > > > >>>
> > > > > https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > > > > > >>> > > [2]
> > > > > > > > > >>> > >
> > > > > > > > > >>>
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > > > > > >>> > >
> > > > > > > > > >>> > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > >>> > > To unsubscribe, e-mail:
> > dev-unsubscribe@kie.apache.org
> > > > > > > > > >>> > > For additional commands, e-mail:
> > > dev-help@kie.apache.org
> > > > > > > > > >>> > >
> > > > > > > > > >>> > >
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > > >>> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > > >>> For additional commands, e-mail:
> dev-help@kie.apache.org
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > >
> > > > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > >
> > > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > For additional commands, e-mail: dev-help@kie.apache.org
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > For additional commands, e-mail: dev-help@kie.apache.org
> >
> >
>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Francisco Javier Tirado Sarti <ft...@redhat.com>]

Hi Martin,
While I analyze the data, let me ask you if it is possible to perform
another check (similar in a way to disabling data-index like you do) Can
you switch to MongoDB persistence and check if the same degradation that is
there for postgres remains?
I do not know if this is feasible but will certainly indicate the problem
is on the postgres storage layer and I do not have a clear prediction of
what we will see when doing this switch.

On Wed, Feb 7, 2024 at 6:37 PM Martin Weiler <mw...@ibm.com.invalid>
wrote:

> Hi Francisco,
>
> thanks for your work on this important topic!
>
> I would like to share some test results here, which might help to improve
> the codebase even further. I am using the jmeter based test case from Pere
> and Enrique (thanks guys!) [1] which uses a load of 30 threads to
>
> 1) start a new process instance (POST)
> 2) retrieve tasks for a user (GET)
> 3) fetches task details (GET)
> 4) complete a task (POST)
> 5) execute a query on data-audit
>
> With this test setup, I noticed that the performance for the POST
> requests, in particular the one to start a new process instance, degrades
> over time - see graph [2]. If I run the same test without data-index, then
> there is no such performance degradation [3]. You can find a thread dump
> captured a few minutes into the first test here [4] that might help to see
> some of the contention points.
>
> I'd appreciate if you could take a look and see if there is something that
> can be further improved based on your previous work. If you need any
> additional data, let me know, but otherwise it is straightforward to run
> the jmeter test as well.
>
> Thanks,
> Martin
>
> [1] https://github.com/pefernan/job-service-refactor-test/
> [2]
> https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
> [3]
> https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
> [4]
> https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing
>
> ________________________________________
> From: Francisco Javier Tirado Sarti <ft...@redhat.com>
> Sent: Wednesday, January 17, 2024 9:13 AM
> To: dev@kie.apache.org
> Cc: Pere Fernandez Perez
> Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with data-index
> persistence addon
>
> Hi Alex,
> I did not take times (which depends on a number of variables that
> drastically change between environments), but verify that the number of
> updates has been reduced drastically without losing functionality, which is
> objectively a good thing. If before the change, for every node executed, we
> have an update for every node previously executed, so if a process have 50
> nodes to execute, we were performing nearly 50*51/2 updates, which gives us
> a total of  1275 updates, now we have just one for every node being
> executed, implying a total of 50 updates.
>
>
> On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli <al...@porcelli.me> wrote:
>
> > Francisco,
> >
> > I noticed that your PR has been merged, but I was expecting (at least
> > was my understanding from this thread) that before merging some
> > benchmark data would be shared in advance - to assess the cost/benefit
> > of such a decent size change.
> >
> > Do you have any information to share?
> >
> > On Sat, Dec 23, 2023 at 4:02 AM Francisco Javier Tirado Sarti
> > <ft...@redhat.com> wrote:
> > >
> > > Yes, as intended, now we have one select and one insert/update per node
> > > event.
> > > I moved the PR as ready for review and give @Pere Fernandez Perez
> > > <pe...@redhat.com> permission to the branch so he can edit it  in
> the
> > > next two weeks (Ill be on PTO)  if desired, before merging.
> > >
> > >
> > > On Thu, Dec 21, 2023 at 5:58 PM Alex Porcelli <al...@porcelli.me>
> wrote:
> > >
> > > > Cool, thank you Francisco!
> > > >
> > > > Did you manage to get some preliminary data about improvements?
> > > >
> > > > On Thu, Dec 21, 2023 at 11:52 AM Francisco Javier Tirado Sarti
> > > > <ft...@redhat.com> wrote:
> > > > >
> > > > > Yes, after some delay because of quarkus 3 migration. Im refining
> > this
> > > > > draft PR
> > > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > > >
> > > > > On Thu, Dec 21, 2023 at 5:48 PM Alex Porcelli <al...@porcelli.me>
> > wrote:
> > > > >
> > > > > > Any update or new findings on this topic?
> > > > > >
> > > > > > On Tue, Nov 28, 2023 at 8:38 AM Francisco Javier Tirado Sarti
> > > > > > <ft...@redhat.com> wrote:
> > > > > > >
> > > > > > > Hi Alex,
> > > > > > > After considering different options to improve performance, we
> > feel
> > > > that
> > > > > > it
> > > > > > > is time to "partially" move away from the current Map style
> > > > interface (
> > > > > > >
> > > > > >
> > > >
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > > )
> > > > > > > which was shared with Trusty, to one more suitable for usage
> > with a
> > > > > > > relational DB like postgresql (but still compatible with big
> > table
> > > > dbs).
> > > > > > > The idea will be to replace generic Storage interface by four
> > > > specific
> > > > > > > interfaces (which will inherit from a common one that keeps the
> > query
> > > > > > part
> > > > > > > at is it. with get and query methods), that will include the
> > required
> > > > > > > modification operations for the four DataIndex "domains":
> > > > > > processinstance,
> > > > > > > usertask, processdefinitions and jobs. Those interfaces will
> > define
> > > > > > methods
> > > > > > > like addNode, addVariable, updateTask, addAttachment..... that
> > will
> > > > allow
> > > > > > > the persistent layer implementation  to just update the needed
> > info
> > > > in
> > > > > > the
> > > > > > > DB  (for example, for addNode in Postgres, just insert a row
> into
> > > > nodes
> > > > > > > table, for addNode in Mongo, basically the same atomic upsert
> > > > operation
> > > > > > > that is currently done). Therefore, we increase performance for
> > > > Postgres
> > > > > > > and keep the current one for Mongo. The current DB schemas
> won't
> > be
> > > > > > > touched.
> > > > > > > Since the code change is large, I do not think I'll be able to
> > have
> > > > the
> > > > > > PR
> > > > > > > ready till next week.
> > > > > > > But before starting, please let me know if that approach is
> fine
> > for
> > > > you.
> > > > > > > Best regards.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Nov 24, 2023 at 6:55 PM Alex Porcelli <
> alex@porcelli.me>
> > > > wrote:
> > > > > > >
> > > > > > > > Thank you Francisco to getting deeper on this…
> > > > > > > >
> > > > > > > > Looking forward to see the results of your suggested
> > improvements.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Nov 24, 2023 at 9:40 AM Francisco Javier Tirado
> Sarti <
> > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > >
> > > > > > > > > I forgot to attach the queries
> > > > > > > > >
> > > > > > > > > On Fri, Nov 24, 2023 at 3:04 PM Francisco Javier Tirado
> > Sarti <
> > > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > > >
> > > > > > > > >> Hi,
> > > > > > > > >> A brief update on this topic.
> > > > > > > > >> After doing a simple test with example
> > > > > > > > >>
> > > > > > > >
> > > > > >
> > > >
> >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > > ,
> > > > > > > > >> the number of updates over Nodes table is n*n, so we
> manage
> > to
> > > > > > obtain a
> > > > > > > > >> perfect quadratic performance degradation. The problem is
> > worse
> > > > in
> > > > > > the
> > > > > > > > case
> > > > > > > > >> of Serverless Workflow than in BPMN because we the number
> of
> > > > nodes
> > > > > > is
> > > > > > > > >> greater than the number of states. In that example N is
> 16,
> > but
> > > > for
> > > > > > a
> > > > > > > > more
> > > > > > > > >> complex workflow it would be certainly large.
> > > > > > > > >> I think that this is more related to how we are handling
> > JPA in
> > > > the
> > > > > > > > code,
> > > > > > > > >> in particular the mapping from model to entity (basically
> > JPA is
> > > > > > blind
> > > > > > > > and
> > > > > > > > >> has to update all nodes for every write because it
> believes
> > the
> > > > > > node has
> > > > > > > > >> been updated, although it is not) than an issue in the
> table
> > > > > > definition.
> > > > > > > > >> In fact, when using JPA, separating the server model from
> > the
> > > > JPA
> > > > > > > > entity is
> > > > > > > > >> not a good idea, especially if the entity contains
> > collections.
> > > > I
> > > > > > will
> > > > > > > > try
> > > > > > > > >> to change that without breaking anything.
> > > > > > > > >>
> > > > > > > > >> On Wed, Nov 22, 2023 at 12:10 PM Enrique Gonzalez
> Martinez <
> > > > > > > > >> egonzalez@apache.org> wrote:
> > > > > > > > >>
> > > > > > > > >>> After the events split you now will need to create a node
> > > > instance
> > > > > > > > >>> model instance of making independent from the process
> > instance.
> > > > > > > > >>> That should do the trick.
> > > > > > > > >>>
> > > > > > > > >>> Regarding deleting/inserting it was fixed at some point.
> > > > > > > > >>>
> > > > > > > > >>> El mar, 21 nov 2023 a las 20:22, Francisco Javier Tirado
> > Sarti
> > > > > > > > >>> (<ft...@redhat.com>) escribió:
> > > > > > > > >>> >
> > > > > > > > >>> > Hi Martin,
> > > > > > > > >>> > I have a task to review performance of
> > > > > > > > >>> >
> > > > > > > > >>> > ProcessInstanceNodeDataEventMerger
> > > > > > > > >>> > My idea is to reduce the number of delete inserts when
> > > > processing
> > > > > > > > >>> events
> > > > > > > > >>> > and try to do it incremental.
> > > > > > > > >>> > That should improve performance.
> > > > > > > > >>> > PS:
> > > > > > > > >>> > I was planning to send an e-mail tomorrow announcing
> > that in
> > > > > > case you
> > > > > > > > >>> were
> > > > > > > > >>> > already working on a fix for that. I assume you are not
> > and I
> > > > > > would
> > > > > > > > be
> > > > > > > > >>> > sending a PR soon.
> > > > > > > > >>> >
> > > > > > > > >>> > On Tue, Nov 21, 2023 at 6:09 PM Martin Weiler
> > > > > > > > <mweiler@ibm.com.invalid
> > > > > > > > >>> >
> > > > > > > > >>> > wrote:
> > > > > > > > >>> >
> > > > > > > > >>> > > I looked into the new examples using data-index
> > persistence
> > > > > > addon -
> > > > > > > > >>> Neus'
> > > > > > > > >>> > > PR#1813 [1] for serverless and Pere's branch [2] for
> > > > workflow
> > > > > > > > (great
> > > > > > > > >>> job
> > > > > > > > >>> > > both!) - and they work without issues using single
> > > > requests.
> > > > > > > > >>> However, under
> > > > > > > > >>> > > some load (I used 'ab' for testing with a light
> > > > concurrency of
> > > > > > 10
> > > > > > > > >>> parallel
> > > > > > > > >>> > > requests) I ran into the following problems:
> > > > > > > > >>> > >
> > > > > > > > >>> > > (1) Large number of insert/delete calls (eg. for
> tables
> > > > such as
> > > > > > > > >>> nodes,
> > > > > > > > >>> > > definitions, etc)
> > > > > > > > >>> > >
> > > > > > > > >>> > > (2) Hibernate OptimisticLockExceptions /
> > > > StaleStateExceptions
> > > > > > > > >>> > >
> > > > > > > > >>> > > (3) DB deadlocks
> > > > > > > > >>> > >
> > > > > > > > >>> > > (4) Error responses, slow response times
> > > > > > > > >>> > >
> > > > > > > > >>> > > The reason I am reaching out with this topic here is
> to
> > > > find
> > > > > > out if
> > > > > > > > >>> we are
> > > > > > > > >>> > > aware of this issue, and if someone is already
> looking
> > > > into or
> > > > > > > > being
> > > > > > > > >>> > > assigned to it?
> > > > > > > > >>> > >
> > > > > > > > >>> > > Thanks,
> > > > > > > > >>> > > Martin
> > > > > > > > >>> > >
> > > > > > > > >>> > > [1]
> > > > > > > > >>>
> > > > https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > > > > >>> > > [2]
> > > > > > > > >>> > >
> > > > > > > > >>>
> > > > > > > >
> > > > > >
> > > >
> >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > > > > >>> > >
> > > > > > > > >>> > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > >>> > > To unsubscribe, e-mail:
> dev-unsubscribe@kie.apache.org
> > > > > > > > >>> > > For additional commands, e-mail:
> > dev-help@kie.apache.org
> > > > > > > > >>> > >
> > > > > > > > >>> > >
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > >>> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > >>> For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > > >
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > >
> > > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > >
> > > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > For additional commands, e-mail: dev-help@kie.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> For additional commands, e-mail: dev-help@kie.apache.org
>
>

RE: [DISCUSSION] Performance issues with data-index persistence addon

[Martin Weiler <mw...@ibm.com.INVALID>]

Hi Francisco,

thanks for your work on this important topic!

I would like to share some test results here, which might help to improve the codebase even further. I am using the jmeter based test case from Pere and Enrique (thanks guys!) [1] which uses a load of 30 threads to

1) start a new process instance (POST)
2) retrieve tasks for a user (GET)
3) fetches task details (GET)
4) complete a task (POST)
5) execute a query on data-audit

With this test setup, I noticed that the performance for the POST requests, in particular the one to start a new process instance, degrades over time - see graph [2]. If I run the same test without data-index, then there is no such performance degradation [3]. You can find a thread dump captured a few minutes into the first test here [4] that might help to see some of the contention points.

I'd appreciate if you could take a look and see if there is something that can be further improved based on your previous work. If you need any additional data, let me know, but otherwise it is straightforward to run the jmeter test as well.

Thanks,
Martin

[1] https://github.com/pefernan/job-service-refactor-test/
[2] https://drive.google.com/file/d/1Gqn-ixE05kXv2jdssAUlnMuUVcHxIYZ0/view?usp=sharing
[3] https://drive.google.com/file/d/10gVNyb4JYg_bA18bNhY9dEDbPn3TOxL7/view?usp=sharing
[4] https://drive.google.com/file/d/1jVrtsO49gCvUlnaC9AUAtkVKTm4PbdUv/view?usp=sharing

________________________________________
From: Francisco Javier Tirado Sarti <ft...@redhat.com>
Sent: Wednesday, January 17, 2024 9:13 AM
To: dev@kie.apache.org
Cc: Pere Fernandez Perez
Subject: [EXTERNAL] Re: [DISCUSSION] Performance issues with data-index persistence addon

Hi Alex,
I did not take times (which depends on a number of variables that
drastically change between environments), but verify that the number of
updates has been reduced drastically without losing functionality, which is
objectively a good thing. If before the change, for every node executed, we
have an update for every node previously executed, so if a process have 50
nodes to execute, we were performing nearly 50*51/2 updates, which gives us
a total of  1275 updates, now we have just one for every node being
executed, implying a total of 50 updates.


On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli <al...@porcelli.me> wrote:

> Francisco,
>
> I noticed that your PR has been merged, but I was expecting (at least
> was my understanding from this thread) that before merging some
> benchmark data would be shared in advance - to assess the cost/benefit
> of such a decent size change.
>
> Do you have any information to share?
>
> On Sat, Dec 23, 2023 at 4:02 AM Francisco Javier Tirado Sarti
> <ft...@redhat.com> wrote:
> >
> > Yes, as intended, now we have one select and one insert/update per node
> > event.
> > I moved the PR as ready for review and give @Pere Fernandez Perez
> > <pe...@redhat.com> permission to the branch so he can edit it  in the
> > next two weeks (Ill be on PTO)  if desired, before merging.
> >
> >
> > On Thu, Dec 21, 2023 at 5:58 PM Alex Porcelli <al...@porcelli.me> wrote:
> >
> > > Cool, thank you Francisco!
> > >
> > > Did you manage to get some preliminary data about improvements?
> > >
> > > On Thu, Dec 21, 2023 at 11:52 AM Francisco Javier Tirado Sarti
> > > <ft...@redhat.com> wrote:
> > > >
> > > > Yes, after some delay because of quarkus 3 migration. Im refining
> this
> > > > draft PR
> > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > >
> > > > On Thu, Dec 21, 2023 at 5:48 PM Alex Porcelli <al...@porcelli.me>
> wrote:
> > > >
> > > > > Any update or new findings on this topic?
> > > > >
> > > > > On Tue, Nov 28, 2023 at 8:38 AM Francisco Javier Tirado Sarti
> > > > > <ft...@redhat.com> wrote:
> > > > > >
> > > > > > Hi Alex,
> > > > > > After considering different options to improve performance, we
> feel
> > > that
> > > > > it
> > > > > > is time to "partially" move away from the current Map style
> > > interface (
> > > > > >
> > > > >
> > >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > )
> > > > > > which was shared with Trusty, to one more suitable for usage
> with a
> > > > > > relational DB like postgresql (but still compatible with big
> table
> > > dbs).
> > > > > > The idea will be to replace generic Storage interface by four
> > > specific
> > > > > > interfaces (which will inherit from a common one that keeps the
> query
> > > > > part
> > > > > > at is it. with get and query methods), that will include the
> required
> > > > > > modification operations for the four DataIndex "domains":
> > > > > processinstance,
> > > > > > usertask, processdefinitions and jobs. Those interfaces will
> define
> > > > > methods
> > > > > > like addNode, addVariable, updateTask, addAttachment..... that
> will
> > > allow
> > > > > > the persistent layer implementation  to just update the needed
> info
> > > in
> > > > > the
> > > > > > DB  (for example, for addNode in Postgres, just insert a row into
> > > nodes
> > > > > > table, for addNode in Mongo, basically the same atomic upsert
> > > operation
> > > > > > that is currently done). Therefore, we increase performance for
> > > Postgres
> > > > > > and keep the current one for Mongo. The current DB schemas won't
> be
> > > > > > touched.
> > > > > > Since the code change is large, I do not think I'll be able to
> have
> > > the
> > > > > PR
> > > > > > ready till next week.
> > > > > > But before starting, please let me know if that approach is fine
> for
> > > you.
> > > > > > Best regards.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Nov 24, 2023 at 6:55 PM Alex Porcelli <al...@porcelli.me>
> > > wrote:
> > > > > >
> > > > > > > Thank you Francisco to getting deeper on this…
> > > > > > >
> > > > > > > Looking forward to see the results of your suggested
> improvements.
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Nov 24, 2023 at 9:40 AM Francisco Javier Tirado Sarti <
> > > > > > > ftirados@redhat.com> wrote:
> > > > > > >
> > > > > > > > I forgot to attach the queries
> > > > > > > >
> > > > > > > > On Fri, Nov 24, 2023 at 3:04 PM Francisco Javier Tirado
> Sarti <
> > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > >
> > > > > > > >> Hi,
> > > > > > > >> A brief update on this topic.
> > > > > > > >> After doing a simple test with example
> > > > > > > >>
> > > > > > >
> > > > >
> > >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > ,
> > > > > > > >> the number of updates over Nodes table is n*n, so we manage
> to
> > > > > obtain a
> > > > > > > >> perfect quadratic performance degradation. The problem is
> worse
> > > in
> > > > > the
> > > > > > > case
> > > > > > > >> of Serverless Workflow than in BPMN because we the number of
> > > nodes
> > > > > is
> > > > > > > >> greater than the number of states. In that example N is 16,
> but
> > > for
> > > > > a
> > > > > > > more
> > > > > > > >> complex workflow it would be certainly large.
> > > > > > > >> I think that this is more related to how we are handling
> JPA in
> > > the
> > > > > > > code,
> > > > > > > >> in particular the mapping from model to entity (basically
> JPA is
> > > > > blind
> > > > > > > and
> > > > > > > >> has to update all nodes for every write because it believes
> the
> > > > > node has
> > > > > > > >> been updated, although it is not) than an issue in the table
> > > > > definition.
> > > > > > > >> In fact, when using JPA, separating the server model from
> the
> > > JPA
> > > > > > > entity is
> > > > > > > >> not a good idea, especially if the entity contains
> collections.
> > > I
> > > > > will
> > > > > > > try
> > > > > > > >> to change that without breaking anything.
> > > > > > > >>
> > > > > > > >> On Wed, Nov 22, 2023 at 12:10 PM Enrique Gonzalez Martinez <
> > > > > > > >> egonzalez@apache.org> wrote:
> > > > > > > >>
> > > > > > > >>> After the events split you now will need to create a node
> > > instance
> > > > > > > >>> model instance of making independent from the process
> instance.
> > > > > > > >>> That should do the trick.
> > > > > > > >>>
> > > > > > > >>> Regarding deleting/inserting it was fixed at some point.
> > > > > > > >>>
> > > > > > > >>> El mar, 21 nov 2023 a las 20:22, Francisco Javier Tirado
> Sarti
> > > > > > > >>> (<ft...@redhat.com>) escribió:
> > > > > > > >>> >
> > > > > > > >>> > Hi Martin,
> > > > > > > >>> > I have a task to review performance of
> > > > > > > >>> >
> > > > > > > >>> > ProcessInstanceNodeDataEventMerger
> > > > > > > >>> > My idea is to reduce the number of delete inserts when
> > > processing
> > > > > > > >>> events
> > > > > > > >>> > and try to do it incremental.
> > > > > > > >>> > That should improve performance.
> > > > > > > >>> > PS:
> > > > > > > >>> > I was planning to send an e-mail tomorrow announcing
> that in
> > > > > case you
> > > > > > > >>> were
> > > > > > > >>> > already working on a fix for that. I assume you are not
> and I
> > > > > would
> > > > > > > be
> > > > > > > >>> > sending a PR soon.
> > > > > > > >>> >
> > > > > > > >>> > On Tue, Nov 21, 2023 at 6:09 PM Martin Weiler
> > > > > > > <mweiler@ibm.com.invalid
> > > > > > > >>> >
> > > > > > > >>> > wrote:
> > > > > > > >>> >
> > > > > > > >>> > > I looked into the new examples using data-index
> persistence
> > > > > addon -
> > > > > > > >>> Neus'
> > > > > > > >>> > > PR#1813 [1] for serverless and Pere's branch [2] for
> > > workflow
> > > > > > > (great
> > > > > > > >>> job
> > > > > > > >>> > > both!) - and they work without issues using single
> > > requests.
> > > > > > > >>> However, under
> > > > > > > >>> > > some load (I used 'ab' for testing with a light
> > > concurrency of
> > > > > 10
> > > > > > > >>> parallel
> > > > > > > >>> > > requests) I ran into the following problems:
> > > > > > > >>> > >
> > > > > > > >>> > > (1) Large number of insert/delete calls (eg. for tables
> > > such as
> > > > > > > >>> nodes,
> > > > > > > >>> > > definitions, etc)
> > > > > > > >>> > >
> > > > > > > >>> > > (2) Hibernate OptimisticLockExceptions /
> > > StaleStateExceptions
> > > > > > > >>> > >
> > > > > > > >>> > > (3) DB deadlocks
> > > > > > > >>> > >
> > > > > > > >>> > > (4) Error responses, slow response times
> > > > > > > >>> > >
> > > > > > > >>> > > The reason I am reaching out with this topic here is to
> > > find
> > > > > out if
> > > > > > > >>> we are
> > > > > > > >>> > > aware of this issue, and if someone is already looking
> > > into or
> > > > > > > being
> > > > > > > >>> > > assigned to it?
> > > > > > > >>> > >
> > > > > > > >>> > > Thanks,
> > > > > > > >>> > > Martin
> > > > > > > >>> > >
> > > > > > > >>> > > [1]
> > > > > > > >>>
> > > https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > > > >>> > > [2]
> > > > > > > >>> > >
> > > > > > > >>>
> > > > > > >
> > > > >
> > >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > > > >>> > >
> > > > > > > >>> > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > >>> > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > >>> > > For additional commands, e-mail:
> dev-help@kie.apache.org
> > > > > > > >>> > >
> > > > > > > >>> > >
> > > > > > > >>>
> > > > > > > >>>
> > > > >
> ---------------------------------------------------------------------
> > > > > > > >>> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > >>> For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > >
> > > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > For additional commands, e-mail: dev-help@kie.apache.org
> > >
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> For additional commands, e-mail: dev-help@kie.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
For additional commands, e-mail: dev-help@kie.apache.org

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Francisco Javier Tirado Sarti <ft...@redhat.com>]

Hi Alex,
I did not take times (which depends on a number of variables that
drastically change between environments), but verify that the number of
updates has been reduced drastically without losing functionality, which is
objectively a good thing. If before the change, for every node executed, we
have an update for every node previously executed, so if a process have 50
nodes to execute, we were performing nearly 50*51/2 updates, which gives us
a total of  1275 updates, now we have just one for every node being
executed, implying a total of 50 updates.


On Wed, Jan 17, 2024 at 3:18 PM Alex Porcelli <al...@porcelli.me> wrote:

> Francisco,
>
> I noticed that your PR has been merged, but I was expecting (at least
> was my understanding from this thread) that before merging some
> benchmark data would be shared in advance - to assess the cost/benefit
> of such a decent size change.
>
> Do you have any information to share?
>
> On Sat, Dec 23, 2023 at 4:02 AM Francisco Javier Tirado Sarti
> <ft...@redhat.com> wrote:
> >
> > Yes, as intended, now we have one select and one insert/update per node
> > event.
> > I moved the PR as ready for review and give @Pere Fernandez Perez
> > <pe...@redhat.com> permission to the branch so he can edit it  in the
> > next two weeks (Ill be on PTO)  if desired, before merging.
> >
> >
> > On Thu, Dec 21, 2023 at 5:58 PM Alex Porcelli <al...@porcelli.me> wrote:
> >
> > > Cool, thank you Francisco!
> > >
> > > Did you manage to get some preliminary data about improvements?
> > >
> > > On Thu, Dec 21, 2023 at 11:52 AM Francisco Javier Tirado Sarti
> > > <ft...@redhat.com> wrote:
> > > >
> > > > Yes, after some delay because of quarkus 3 migration. Im refining
> this
> > > > draft PR
> > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > > >
> > > > On Thu, Dec 21, 2023 at 5:48 PM Alex Porcelli <al...@porcelli.me>
> wrote:
> > > >
> > > > > Any update or new findings on this topic?
> > > > >
> > > > > On Tue, Nov 28, 2023 at 8:38 AM Francisco Javier Tirado Sarti
> > > > > <ft...@redhat.com> wrote:
> > > > > >
> > > > > > Hi Alex,
> > > > > > After considering different options to improve performance, we
> feel
> > > that
> > > > > it
> > > > > > is time to "partially" move away from the current Map style
> > > interface (
> > > > > >
> > > > >
> > >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > > )
> > > > > > which was shared with Trusty, to one more suitable for usage
> with a
> > > > > > relational DB like postgresql (but still compatible with big
> table
> > > dbs).
> > > > > > The idea will be to replace generic Storage interface by four
> > > specific
> > > > > > interfaces (which will inherit from a common one that keeps the
> query
> > > > > part
> > > > > > at is it. with get and query methods), that will include the
> required
> > > > > > modification operations for the four DataIndex "domains":
> > > > > processinstance,
> > > > > > usertask, processdefinitions and jobs. Those interfaces will
> define
> > > > > methods
> > > > > > like addNode, addVariable, updateTask, addAttachment..... that
> will
> > > allow
> > > > > > the persistent layer implementation  to just update the needed
> info
> > > in
> > > > > the
> > > > > > DB  (for example, for addNode in Postgres, just insert a row into
> > > nodes
> > > > > > table, for addNode in Mongo, basically the same atomic upsert
> > > operation
> > > > > > that is currently done). Therefore, we increase performance for
> > > Postgres
> > > > > > and keep the current one for Mongo. The current DB schemas won't
> be
> > > > > > touched.
> > > > > > Since the code change is large, I do not think I'll be able to
> have
> > > the
> > > > > PR
> > > > > > ready till next week.
> > > > > > But before starting, please let me know if that approach is fine
> for
> > > you.
> > > > > > Best regards.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Nov 24, 2023 at 6:55 PM Alex Porcelli <al...@porcelli.me>
> > > wrote:
> > > > > >
> > > > > > > Thank you Francisco to getting deeper on this…
> > > > > > >
> > > > > > > Looking forward to see the results of your suggested
> improvements.
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Nov 24, 2023 at 9:40 AM Francisco Javier Tirado Sarti <
> > > > > > > ftirados@redhat.com> wrote:
> > > > > > >
> > > > > > > > I forgot to attach the queries
> > > > > > > >
> > > > > > > > On Fri, Nov 24, 2023 at 3:04 PM Francisco Javier Tirado
> Sarti <
> > > > > > > > ftirados@redhat.com> wrote:
> > > > > > > >
> > > > > > > >> Hi,
> > > > > > > >> A brief update on this topic.
> > > > > > > >> After doing a simple test with example
> > > > > > > >>
> > > > > > >
> > > > >
> > >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > > ,
> > > > > > > >> the number of updates over Nodes table is n*n, so we manage
> to
> > > > > obtain a
> > > > > > > >> perfect quadratic performance degradation. The problem is
> worse
> > > in
> > > > > the
> > > > > > > case
> > > > > > > >> of Serverless Workflow than in BPMN because we the number of
> > > nodes
> > > > > is
> > > > > > > >> greater than the number of states. In that example N is 16,
> but
> > > for
> > > > > a
> > > > > > > more
> > > > > > > >> complex workflow it would be certainly large.
> > > > > > > >> I think that this is more related to how we are handling
> JPA in
> > > the
> > > > > > > code,
> > > > > > > >> in particular the mapping from model to entity (basically
> JPA is
> > > > > blind
> > > > > > > and
> > > > > > > >> has to update all nodes for every write because it believes
> the
> > > > > node has
> > > > > > > >> been updated, although it is not) than an issue in the table
> > > > > definition.
> > > > > > > >> In fact, when using JPA, separating the server model from
> the
> > > JPA
> > > > > > > entity is
> > > > > > > >> not a good idea, especially if the entity contains
> collections.
> > > I
> > > > > will
> > > > > > > try
> > > > > > > >> to change that without breaking anything.
> > > > > > > >>
> > > > > > > >> On Wed, Nov 22, 2023 at 12:10 PM Enrique Gonzalez Martinez <
> > > > > > > >> egonzalez@apache.org> wrote:
> > > > > > > >>
> > > > > > > >>> After the events split you now will need to create a node
> > > instance
> > > > > > > >>> model instance of making independent from the process
> instance.
> > > > > > > >>> That should do the trick.
> > > > > > > >>>
> > > > > > > >>> Regarding deleting/inserting it was fixed at some point.
> > > > > > > >>>
> > > > > > > >>> El mar, 21 nov 2023 a las 20:22, Francisco Javier Tirado
> Sarti
> > > > > > > >>> (<ft...@redhat.com>) escribió:
> > > > > > > >>> >
> > > > > > > >>> > Hi Martin,
> > > > > > > >>> > I have a task to review performance of
> > > > > > > >>> >
> > > > > > > >>> > ProcessInstanceNodeDataEventMerger
> > > > > > > >>> > My idea is to reduce the number of delete inserts when
> > > processing
> > > > > > > >>> events
> > > > > > > >>> > and try to do it incremental.
> > > > > > > >>> > That should improve performance.
> > > > > > > >>> > PS:
> > > > > > > >>> > I was planning to send an e-mail tomorrow announcing
> that in
> > > > > case you
> > > > > > > >>> were
> > > > > > > >>> > already working on a fix for that. I assume you are not
> and I
> > > > > would
> > > > > > > be
> > > > > > > >>> > sending a PR soon.
> > > > > > > >>> >
> > > > > > > >>> > On Tue, Nov 21, 2023 at 6:09 PM Martin Weiler
> > > > > > > <mweiler@ibm.com.invalid
> > > > > > > >>> >
> > > > > > > >>> > wrote:
> > > > > > > >>> >
> > > > > > > >>> > > I looked into the new examples using data-index
> persistence
> > > > > addon -
> > > > > > > >>> Neus'
> > > > > > > >>> > > PR#1813 [1] for serverless and Pere's branch [2] for
> > > workflow
> > > > > > > (great
> > > > > > > >>> job
> > > > > > > >>> > > both!) - and they work without issues using single
> > > requests.
> > > > > > > >>> However, under
> > > > > > > >>> > > some load (I used 'ab' for testing with a light
> > > concurrency of
> > > > > 10
> > > > > > > >>> parallel
> > > > > > > >>> > > requests) I ran into the following problems:
> > > > > > > >>> > >
> > > > > > > >>> > > (1) Large number of insert/delete calls (eg. for tables
> > > such as
> > > > > > > >>> nodes,
> > > > > > > >>> > > definitions, etc)
> > > > > > > >>> > >
> > > > > > > >>> > > (2) Hibernate OptimisticLockExceptions /
> > > StaleStateExceptions
> > > > > > > >>> > >
> > > > > > > >>> > > (3) DB deadlocks
> > > > > > > >>> > >
> > > > > > > >>> > > (4) Error responses, slow response times
> > > > > > > >>> > >
> > > > > > > >>> > > The reason I am reaching out with this topic here is to
> > > find
> > > > > out if
> > > > > > > >>> we are
> > > > > > > >>> > > aware of this issue, and if someone is already looking
> > > into or
> > > > > > > being
> > > > > > > >>> > > assigned to it?
> > > > > > > >>> > >
> > > > > > > >>> > > Thanks,
> > > > > > > >>> > > Martin
> > > > > > > >>> > >
> > > > > > > >>> > > [1]
> > > > > > > >>>
> > > https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > > > >>> > > [2]
> > > > > > > >>> > >
> > > > > > > >>>
> > > > > > >
> > > > >
> > >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > > > >>> > >
> > > > > > > >>> > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > >>> > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > >>> > > For additional commands, e-mail:
> dev-help@kie.apache.org
> > > > > > > >>> > >
> > > > > > > >>> > >
> > > > > > > >>>
> > > > > > > >>>
> > > > >
> ---------------------------------------------------------------------
> > > > > > > >>> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > >>> For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > >
> > > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > For additional commands, e-mail: dev-help@kie.apache.org
> > >
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> For additional commands, e-mail: dev-help@kie.apache.org
>
>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Alex Porcelli <al...@porcelli.me>]

Francisco,

I noticed that your PR has been merged, but I was expecting (at least
was my understanding from this thread) that before merging some
benchmark data would be shared in advance - to assess the cost/benefit
of such a decent size change.

Do you have any information to share?

On Sat, Dec 23, 2023 at 4:02 AM Francisco Javier Tirado Sarti
<ft...@redhat.com> wrote:
>
> Yes, as intended, now we have one select and one insert/update per node
> event.
> I moved the PR as ready for review and give @Pere Fernandez Perez
> <pe...@redhat.com> permission to the branch so he can edit it  in the
> next two weeks (Ill be on PTO)  if desired, before merging.
>
>
> On Thu, Dec 21, 2023 at 5:58 PM Alex Porcelli <al...@porcelli.me> wrote:
>
> > Cool, thank you Francisco!
> >
> > Did you manage to get some preliminary data about improvements?
> >
> > On Thu, Dec 21, 2023 at 11:52 AM Francisco Javier Tirado Sarti
> > <ft...@redhat.com> wrote:
> > >
> > > Yes, after some delay because of quarkus 3 migration. Im refining this
> > > draft PR
> > > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> > >
> > > On Thu, Dec 21, 2023 at 5:48 PM Alex Porcelli <al...@porcelli.me> wrote:
> > >
> > > > Any update or new findings on this topic?
> > > >
> > > > On Tue, Nov 28, 2023 at 8:38 AM Francisco Javier Tirado Sarti
> > > > <ft...@redhat.com> wrote:
> > > > >
> > > > > Hi Alex,
> > > > > After considering different options to improve performance, we feel
> > that
> > > > it
> > > > > is time to "partially" move away from the current Map style
> > interface (
> > > > >
> > > >
> > https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > > )
> > > > > which was shared with Trusty, to one more suitable for usage with a
> > > > > relational DB like postgresql (but still compatible with big table
> > dbs).
> > > > > The idea will be to replace generic Storage interface by four
> > specific
> > > > > interfaces (which will inherit from a common one that keeps the query
> > > > part
> > > > > at is it. with get and query methods), that will include the required
> > > > > modification operations for the four DataIndex "domains":
> > > > processinstance,
> > > > > usertask, processdefinitions and jobs. Those interfaces will define
> > > > methods
> > > > > like addNode, addVariable, updateTask, addAttachment..... that will
> > allow
> > > > > the persistent layer implementation  to just update the needed info
> > in
> > > > the
> > > > > DB  (for example, for addNode in Postgres, just insert a row into
> > nodes
> > > > > table, for addNode in Mongo, basically the same atomic upsert
> > operation
> > > > > that is currently done). Therefore, we increase performance for
> > Postgres
> > > > > and keep the current one for Mongo. The current DB schemas won't be
> > > > > touched.
> > > > > Since the code change is large, I do not think I'll be able to have
> > the
> > > > PR
> > > > > ready till next week.
> > > > > But before starting, please let me know if that approach is fine for
> > you.
> > > > > Best regards.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Nov 24, 2023 at 6:55 PM Alex Porcelli <al...@porcelli.me>
> > wrote:
> > > > >
> > > > > > Thank you Francisco to getting deeper on this…
> > > > > >
> > > > > > Looking forward to see the results of your suggested improvements.
> > > > > >
> > > > > >
> > > > > > On Fri, Nov 24, 2023 at 9:40 AM Francisco Javier Tirado Sarti <
> > > > > > ftirados@redhat.com> wrote:
> > > > > >
> > > > > > > I forgot to attach the queries
> > > > > > >
> > > > > > > On Fri, Nov 24, 2023 at 3:04 PM Francisco Javier Tirado Sarti <
> > > > > > > ftirados@redhat.com> wrote:
> > > > > > >
> > > > > > >> Hi,
> > > > > > >> A brief update on this topic.
> > > > > > >> After doing a simple test with example
> > > > > > >>
> > > > > >
> > > >
> > https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > > ,
> > > > > > >> the number of updates over Nodes table is n*n, so we manage to
> > > > obtain a
> > > > > > >> perfect quadratic performance degradation. The problem is worse
> > in
> > > > the
> > > > > > case
> > > > > > >> of Serverless Workflow than in BPMN because we the number of
> > nodes
> > > > is
> > > > > > >> greater than the number of states. In that example N is 16, but
> > for
> > > > a
> > > > > > more
> > > > > > >> complex workflow it would be certainly large.
> > > > > > >> I think that this is more related to how we are handling JPA in
> > the
> > > > > > code,
> > > > > > >> in particular the mapping from model to entity (basically JPA is
> > > > blind
> > > > > > and
> > > > > > >> has to update all nodes for every write because it believes the
> > > > node has
> > > > > > >> been updated, although it is not) than an issue in the table
> > > > definition.
> > > > > > >> In fact, when using JPA, separating the server model from the
> > JPA
> > > > > > entity is
> > > > > > >> not a good idea, especially if the entity contains collections.
> > I
> > > > will
> > > > > > try
> > > > > > >> to change that without breaking anything.
> > > > > > >>
> > > > > > >> On Wed, Nov 22, 2023 at 12:10 PM Enrique Gonzalez Martinez <
> > > > > > >> egonzalez@apache.org> wrote:
> > > > > > >>
> > > > > > >>> After the events split you now will need to create a node
> > instance
> > > > > > >>> model instance of making independent from the process instance.
> > > > > > >>> That should do the trick.
> > > > > > >>>
> > > > > > >>> Regarding deleting/inserting it was fixed at some point.
> > > > > > >>>
> > > > > > >>> El mar, 21 nov 2023 a las 20:22, Francisco Javier Tirado Sarti
> > > > > > >>> (<ft...@redhat.com>) escribió:
> > > > > > >>> >
> > > > > > >>> > Hi Martin,
> > > > > > >>> > I have a task to review performance of
> > > > > > >>> >
> > > > > > >>> > ProcessInstanceNodeDataEventMerger
> > > > > > >>> > My idea is to reduce the number of delete inserts when
> > processing
> > > > > > >>> events
> > > > > > >>> > and try to do it incremental.
> > > > > > >>> > That should improve performance.
> > > > > > >>> > PS:
> > > > > > >>> > I was planning to send an e-mail tomorrow announcing that in
> > > > case you
> > > > > > >>> were
> > > > > > >>> > already working on a fix for that. I assume you are not and I
> > > > would
> > > > > > be
> > > > > > >>> > sending a PR soon.
> > > > > > >>> >
> > > > > > >>> > On Tue, Nov 21, 2023 at 6:09 PM Martin Weiler
> > > > > > <mweiler@ibm.com.invalid
> > > > > > >>> >
> > > > > > >>> > wrote:
> > > > > > >>> >
> > > > > > >>> > > I looked into the new examples using data-index persistence
> > > > addon -
> > > > > > >>> Neus'
> > > > > > >>> > > PR#1813 [1] for serverless and Pere's branch [2] for
> > workflow
> > > > > > (great
> > > > > > >>> job
> > > > > > >>> > > both!) - and they work without issues using single
> > requests.
> > > > > > >>> However, under
> > > > > > >>> > > some load (I used 'ab' for testing with a light
> > concurrency of
> > > > 10
> > > > > > >>> parallel
> > > > > > >>> > > requests) I ran into the following problems:
> > > > > > >>> > >
> > > > > > >>> > > (1) Large number of insert/delete calls (eg. for tables
> > such as
> > > > > > >>> nodes,
> > > > > > >>> > > definitions, etc)
> > > > > > >>> > >
> > > > > > >>> > > (2) Hibernate OptimisticLockExceptions /
> > StaleStateExceptions
> > > > > > >>> > >
> > > > > > >>> > > (3) DB deadlocks
> > > > > > >>> > >
> > > > > > >>> > > (4) Error responses, slow response times
> > > > > > >>> > >
> > > > > > >>> > > The reason I am reaching out with this topic here is to
> > find
> > > > out if
> > > > > > >>> we are
> > > > > > >>> > > aware of this issue, and if someone is already looking
> > into or
> > > > > > being
> > > > > > >>> > > assigned to it?
> > > > > > >>> > >
> > > > > > >>> > > Thanks,
> > > > > > >>> > > Martin
> > > > > > >>> > >
> > > > > > >>> > > [1]
> > > > > > >>>
> > https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > > >>> > > [2]
> > > > > > >>> > >
> > > > > > >>>
> > > > > >
> > > >
> > https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > > >>> > >
> > > > > > >>> > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > >>> > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > >>> > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > >>> > >
> > > > > > >>> > >
> > > > > > >>>
> > > > > > >>>
> > > > ---------------------------------------------------------------------
> > > > > > >>> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > >>> For additional commands, e-mail: dev-help@kie.apache.org
> > > > > > >>>
> > > > > > >>>
> > > > > > >
> > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > >
> > > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > For additional commands, e-mail: dev-help@kie.apache.org
> >
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
For additional commands, e-mail: dev-help@kie.apache.org

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Francisco Javier Tirado Sarti <ft...@redhat.com>]

Yes, as intended, now we have one select and one insert/update per node
event.
I moved the PR as ready for review and give @Pere Fernandez Perez
<pe...@redhat.com> permission to the branch so he can edit it  in the
next two weeks (Ill be on PTO)  if desired, before merging.


On Thu, Dec 21, 2023 at 5:58 PM Alex Porcelli <al...@porcelli.me> wrote:

> Cool, thank you Francisco!
>
> Did you manage to get some preliminary data about improvements?
>
> On Thu, Dec 21, 2023 at 11:52 AM Francisco Javier Tirado Sarti
> <ft...@redhat.com> wrote:
> >
> > Yes, after some delay because of quarkus 3 migration. Im refining this
> > draft PR
> > https://github.com/apache/incubator-kie-kogito-apps/pull/1941
> >
> > On Thu, Dec 21, 2023 at 5:48 PM Alex Porcelli <al...@porcelli.me> wrote:
> >
> > > Any update or new findings on this topic?
> > >
> > > On Tue, Nov 28, 2023 at 8:38 AM Francisco Javier Tirado Sarti
> > > <ft...@redhat.com> wrote:
> > > >
> > > > Hi Alex,
> > > > After considering different options to improve performance, we feel
> that
> > > it
> > > > is time to "partially" move away from the current Map style
> interface (
> > > >
> > >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > > )
> > > > which was shared with Trusty, to one more suitable for usage with a
> > > > relational DB like postgresql (but still compatible with big table
> dbs).
> > > > The idea will be to replace generic Storage interface by four
> specific
> > > > interfaces (which will inherit from a common one that keeps the query
> > > part
> > > > at is it. with get and query methods), that will include the required
> > > > modification operations for the four DataIndex "domains":
> > > processinstance,
> > > > usertask, processdefinitions and jobs. Those interfaces will define
> > > methods
> > > > like addNode, addVariable, updateTask, addAttachment..... that will
> allow
> > > > the persistent layer implementation  to just update the needed info
> in
> > > the
> > > > DB  (for example, for addNode in Postgres, just insert a row into
> nodes
> > > > table, for addNode in Mongo, basically the same atomic upsert
> operation
> > > > that is currently done). Therefore, we increase performance for
> Postgres
> > > > and keep the current one for Mongo. The current DB schemas won't be
> > > > touched.
> > > > Since the code change is large, I do not think I'll be able to have
> the
> > > PR
> > > > ready till next week.
> > > > But before starting, please let me know if that approach is fine for
> you.
> > > > Best regards.
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, Nov 24, 2023 at 6:55 PM Alex Porcelli <al...@porcelli.me>
> wrote:
> > > >
> > > > > Thank you Francisco to getting deeper on this…
> > > > >
> > > > > Looking forward to see the results of your suggested improvements.
> > > > >
> > > > >
> > > > > On Fri, Nov 24, 2023 at 9:40 AM Francisco Javier Tirado Sarti <
> > > > > ftirados@redhat.com> wrote:
> > > > >
> > > > > > I forgot to attach the queries
> > > > > >
> > > > > > On Fri, Nov 24, 2023 at 3:04 PM Francisco Javier Tirado Sarti <
> > > > > > ftirados@redhat.com> wrote:
> > > > > >
> > > > > >> Hi,
> > > > > >> A brief update on this topic.
> > > > > >> After doing a simple test with example
> > > > > >>
> > > > >
> > >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > > ,
> > > > > >> the number of updates over Nodes table is n*n, so we manage to
> > > obtain a
> > > > > >> perfect quadratic performance degradation. The problem is worse
> in
> > > the
> > > > > case
> > > > > >> of Serverless Workflow than in BPMN because we the number of
> nodes
> > > is
> > > > > >> greater than the number of states. In that example N is 16, but
> for
> > > a
> > > > > more
> > > > > >> complex workflow it would be certainly large.
> > > > > >> I think that this is more related to how we are handling JPA in
> the
> > > > > code,
> > > > > >> in particular the mapping from model to entity (basically JPA is
> > > blind
> > > > > and
> > > > > >> has to update all nodes for every write because it believes the
> > > node has
> > > > > >> been updated, although it is not) than an issue in the table
> > > definition.
> > > > > >> In fact, when using JPA, separating the server model from the
> JPA
> > > > > entity is
> > > > > >> not a good idea, especially if the entity contains collections.
> I
> > > will
> > > > > try
> > > > > >> to change that without breaking anything.
> > > > > >>
> > > > > >> On Wed, Nov 22, 2023 at 12:10 PM Enrique Gonzalez Martinez <
> > > > > >> egonzalez@apache.org> wrote:
> > > > > >>
> > > > > >>> After the events split you now will need to create a node
> instance
> > > > > >>> model instance of making independent from the process instance.
> > > > > >>> That should do the trick.
> > > > > >>>
> > > > > >>> Regarding deleting/inserting it was fixed at some point.
> > > > > >>>
> > > > > >>> El mar, 21 nov 2023 a las 20:22, Francisco Javier Tirado Sarti
> > > > > >>> (<ft...@redhat.com>) escribió:
> > > > > >>> >
> > > > > >>> > Hi Martin,
> > > > > >>> > I have a task to review performance of
> > > > > >>> >
> > > > > >>> > ProcessInstanceNodeDataEventMerger
> > > > > >>> > My idea is to reduce the number of delete inserts when
> processing
> > > > > >>> events
> > > > > >>> > and try to do it incremental.
> > > > > >>> > That should improve performance.
> > > > > >>> > PS:
> > > > > >>> > I was planning to send an e-mail tomorrow announcing that in
> > > case you
> > > > > >>> were
> > > > > >>> > already working on a fix for that. I assume you are not and I
> > > would
> > > > > be
> > > > > >>> > sending a PR soon.
> > > > > >>> >
> > > > > >>> > On Tue, Nov 21, 2023 at 6:09 PM Martin Weiler
> > > > > <mweiler@ibm.com.invalid
> > > > > >>> >
> > > > > >>> > wrote:
> > > > > >>> >
> > > > > >>> > > I looked into the new examples using data-index persistence
> > > addon -
> > > > > >>> Neus'
> > > > > >>> > > PR#1813 [1] for serverless and Pere's branch [2] for
> workflow
> > > > > (great
> > > > > >>> job
> > > > > >>> > > both!) - and they work without issues using single
> requests.
> > > > > >>> However, under
> > > > > >>> > > some load (I used 'ab' for testing with a light
> concurrency of
> > > 10
> > > > > >>> parallel
> > > > > >>> > > requests) I ran into the following problems:
> > > > > >>> > >
> > > > > >>> > > (1) Large number of insert/delete calls (eg. for tables
> such as
> > > > > >>> nodes,
> > > > > >>> > > definitions, etc)
> > > > > >>> > >
> > > > > >>> > > (2) Hibernate OptimisticLockExceptions /
> StaleStateExceptions
> > > > > >>> > >
> > > > > >>> > > (3) DB deadlocks
> > > > > >>> > >
> > > > > >>> > > (4) Error responses, slow response times
> > > > > >>> > >
> > > > > >>> > > The reason I am reaching out with this topic here is to
> find
> > > out if
> > > > > >>> we are
> > > > > >>> > > aware of this issue, and if someone is already looking
> into or
> > > > > being
> > > > > >>> > > assigned to it?
> > > > > >>> > >
> > > > > >>> > > Thanks,
> > > > > >>> > > Martin
> > > > > >>> > >
> > > > > >>> > > [1]
> > > > > >>>
> https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > > >>> > > [2]
> > > > > >>> > >
> > > > > >>>
> > > > >
> > >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > > >>> > >
> > > > > >>> > >
> > > > >
> ---------------------------------------------------------------------
> > > > > >>> > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > >>> > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > > >>> > >
> > > > > >>> > >
> > > > > >>>
> > > > > >>>
> > > ---------------------------------------------------------------------
> > > > > >>> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > >>> For additional commands, e-mail: dev-help@kie.apache.org
> > > > > >>>
> > > > > >>>
> > > > > >
> ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > For additional commands, e-mail: dev-help@kie.apache.org
> > >
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> For additional commands, e-mail: dev-help@kie.apache.org
>
>

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Alex Porcelli <al...@porcelli.me>]

Cool, thank you Francisco!

Did you manage to get some preliminary data about improvements?

On Thu, Dec 21, 2023 at 11:52 AM Francisco Javier Tirado Sarti
<ft...@redhat.com> wrote:
>
> Yes, after some delay because of quarkus 3 migration. Im refining this
> draft PR
> https://github.com/apache/incubator-kie-kogito-apps/pull/1941
>
> On Thu, Dec 21, 2023 at 5:48 PM Alex Porcelli <al...@porcelli.me> wrote:
>
> > Any update or new findings on this topic?
> >
> > On Tue, Nov 28, 2023 at 8:38 AM Francisco Javier Tirado Sarti
> > <ft...@redhat.com> wrote:
> > >
> > > Hi Alex,
> > > After considering different options to improve performance, we feel that
> > it
> > > is time to "partially" move away from the current Map style interface (
> > >
> > https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> > )
> > > which was shared with Trusty, to one more suitable for usage with a
> > > relational DB like postgresql (but still compatible with big table dbs).
> > > The idea will be to replace generic Storage interface by four specific
> > > interfaces (which will inherit from a common one that keeps the query
> > part
> > > at is it. with get and query methods), that will include the required
> > > modification operations for the four DataIndex "domains":
> > processinstance,
> > > usertask, processdefinitions and jobs. Those interfaces will define
> > methods
> > > like addNode, addVariable, updateTask, addAttachment..... that will allow
> > > the persistent layer implementation  to just update the needed info in
> > the
> > > DB  (for example, for addNode in Postgres, just insert a row into nodes
> > > table, for addNode in Mongo, basically the same atomic upsert operation
> > > that is currently done). Therefore, we increase performance for Postgres
> > > and keep the current one for Mongo. The current DB schemas won't be
> > > touched.
> > > Since the code change is large, I do not think I'll be able to have the
> > PR
> > > ready till next week.
> > > But before starting, please let me know if that approach is fine for you.
> > > Best regards.
> > >
> > >
> > >
> > >
> > > On Fri, Nov 24, 2023 at 6:55 PM Alex Porcelli <al...@porcelli.me> wrote:
> > >
> > > > Thank you Francisco to getting deeper on this…
> > > >
> > > > Looking forward to see the results of your suggested improvements.
> > > >
> > > >
> > > > On Fri, Nov 24, 2023 at 9:40 AM Francisco Javier Tirado Sarti <
> > > > ftirados@redhat.com> wrote:
> > > >
> > > > > I forgot to attach the queries
> > > > >
> > > > > On Fri, Nov 24, 2023 at 3:04 PM Francisco Javier Tirado Sarti <
> > > > > ftirados@redhat.com> wrote:
> > > > >
> > > > >> Hi,
> > > > >> A brief update on this topic.
> > > > >> After doing a simple test with example
> > > > >>
> > > >
> > https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > > ,
> > > > >> the number of updates over Nodes table is n*n, so we manage to
> > obtain a
> > > > >> perfect quadratic performance degradation. The problem is worse in
> > the
> > > > case
> > > > >> of Serverless Workflow than in BPMN because we the number of nodes
> > is
> > > > >> greater than the number of states. In that example N is 16, but for
> > a
> > > > more
> > > > >> complex workflow it would be certainly large.
> > > > >> I think that this is more related to how we are handling JPA in the
> > > > code,
> > > > >> in particular the mapping from model to entity (basically JPA is
> > blind
> > > > and
> > > > >> has to update all nodes for every write because it believes the
> > node has
> > > > >> been updated, although it is not) than an issue in the table
> > definition.
> > > > >> In fact, when using JPA, separating the server model from the JPA
> > > > entity is
> > > > >> not a good idea, especially if the entity contains collections. I
> > will
> > > > try
> > > > >> to change that without breaking anything.
> > > > >>
> > > > >> On Wed, Nov 22, 2023 at 12:10 PM Enrique Gonzalez Martinez <
> > > > >> egonzalez@apache.org> wrote:
> > > > >>
> > > > >>> After the events split you now will need to create a node instance
> > > > >>> model instance of making independent from the process instance.
> > > > >>> That should do the trick.
> > > > >>>
> > > > >>> Regarding deleting/inserting it was fixed at some point.
> > > > >>>
> > > > >>> El mar, 21 nov 2023 a las 20:22, Francisco Javier Tirado Sarti
> > > > >>> (<ft...@redhat.com>) escribió:
> > > > >>> >
> > > > >>> > Hi Martin,
> > > > >>> > I have a task to review performance of
> > > > >>> >
> > > > >>> > ProcessInstanceNodeDataEventMerger
> > > > >>> > My idea is to reduce the number of delete inserts when processing
> > > > >>> events
> > > > >>> > and try to do it incremental.
> > > > >>> > That should improve performance.
> > > > >>> > PS:
> > > > >>> > I was planning to send an e-mail tomorrow announcing that in
> > case you
> > > > >>> were
> > > > >>> > already working on a fix for that. I assume you are not and I
> > would
> > > > be
> > > > >>> > sending a PR soon.
> > > > >>> >
> > > > >>> > On Tue, Nov 21, 2023 at 6:09 PM Martin Weiler
> > > > <mweiler@ibm.com.invalid
> > > > >>> >
> > > > >>> > wrote:
> > > > >>> >
> > > > >>> > > I looked into the new examples using data-index persistence
> > addon -
> > > > >>> Neus'
> > > > >>> > > PR#1813 [1] for serverless and Pere's branch [2] for workflow
> > > > (great
> > > > >>> job
> > > > >>> > > both!) - and they work without issues using single requests.
> > > > >>> However, under
> > > > >>> > > some load (I used 'ab' for testing with a light concurrency of
> > 10
> > > > >>> parallel
> > > > >>> > > requests) I ran into the following problems:
> > > > >>> > >
> > > > >>> > > (1) Large number of insert/delete calls (eg. for tables such as
> > > > >>> nodes,
> > > > >>> > > definitions, etc)
> > > > >>> > >
> > > > >>> > > (2) Hibernate OptimisticLockExceptions / StaleStateExceptions
> > > > >>> > >
> > > > >>> > > (3) DB deadlocks
> > > > >>> > >
> > > > >>> > > (4) Error responses, slow response times
> > > > >>> > >
> > > > >>> > > The reason I am reaching out with this topic here is to find
> > out if
> > > > >>> we are
> > > > >>> > > aware of this issue, and if someone is already looking into or
> > > > being
> > > > >>> > > assigned to it?
> > > > >>> > >
> > > > >>> > > Thanks,
> > > > >>> > > Martin
> > > > >>> > >
> > > > >>> > > [1]
> > > > >>> https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > > >>> > > [2]
> > > > >>> > >
> > > > >>>
> > > >
> > https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > > >>> > >
> > > > >>> > >
> > > > ---------------------------------------------------------------------
> > > > >>> > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > >>> > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > >>> > >
> > > > >>> > >
> > > > >>>
> > > > >>>
> > ---------------------------------------------------------------------
> > > > >>> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > >>> For additional commands, e-mail: dev-help@kie.apache.org
> > > > >>>
> > > > >>>
> > > > > ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > For additional commands, e-mail: dev-help@kie.apache.org
> >
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
For additional commands, e-mail: dev-help@kie.apache.org

Re: [DISCUSSION] Performance issues with data-index persistence addon

[Francisco Javier Tirado Sarti <ft...@redhat.com>]

Yes, after some delay because of quarkus 3 migration. Im refining this
draft PR
https://github.com/apache/incubator-kie-kogito-apps/pull/1941

On Thu, Dec 21, 2023 at 5:48 PM Alex Porcelli <al...@porcelli.me> wrote:

> Any update or new findings on this topic?
>
> On Tue, Nov 28, 2023 at 8:38 AM Francisco Javier Tirado Sarti
> <ft...@redhat.com> wrote:
> >
> > Hi Alex,
> > After considering different options to improve performance, we feel that
> it
> > is time to "partially" move away from the current Map style interface (
> >
> https://github.com/apache/incubator-kie-kogito-apps/blob/main/persistence-commons/persistence-commons-api/src/main/java/org/kie/kogito/persistence/api/Storage.java
> )
> > which was shared with Trusty, to one more suitable for usage with a
> > relational DB like postgresql (but still compatible with big table dbs).
> > The idea will be to replace generic Storage interface by four specific
> > interfaces (which will inherit from a common one that keeps the query
> part
> > at is it. with get and query methods), that will include the required
> > modification operations for the four DataIndex "domains":
> processinstance,
> > usertask, processdefinitions and jobs. Those interfaces will define
> methods
> > like addNode, addVariable, updateTask, addAttachment..... that will allow
> > the persistent layer implementation  to just update the needed info in
> the
> > DB  (for example, for addNode in Postgres, just insert a row into nodes
> > table, for addNode in Mongo, basically the same atomic upsert operation
> > that is currently done). Therefore, we increase performance for Postgres
> > and keep the current one for Mongo. The current DB schemas won't be
> > touched.
> > Since the code change is large, I do not think I'll be able to have the
> PR
> > ready till next week.
> > But before starting, please let me know if that approach is fine for you.
> > Best regards.
> >
> >
> >
> >
> > On Fri, Nov 24, 2023 at 6:55 PM Alex Porcelli <al...@porcelli.me> wrote:
> >
> > > Thank you Francisco to getting deeper on this…
> > >
> > > Looking forward to see the results of your suggested improvements.
> > >
> > >
> > > On Fri, Nov 24, 2023 at 9:40 AM Francisco Javier Tirado Sarti <
> > > ftirados@redhat.com> wrote:
> > >
> > > > I forgot to attach the queries
> > > >
> > > > On Fri, Nov 24, 2023 at 3:04 PM Francisco Javier Tirado Sarti <
> > > > ftirados@redhat.com> wrote:
> > > >
> > > >> Hi,
> > > >> A brief update on this topic.
> > > >> After doing a simple test with example
> > > >>
> > >
> https://github.com/apache/incubator-kie-kogito-examples/tree/stable/serverless-workflow-examples/serverless-workflow-data-index-quarkus
> > > ,
> > > >> the number of updates over Nodes table is n*n, so we manage to
> obtain a
> > > >> perfect quadratic performance degradation. The problem is worse in
> the
> > > case
> > > >> of Serverless Workflow than in BPMN because we the number of nodes
> is
> > > >> greater than the number of states. In that example N is 16, but for
> a
> > > more
> > > >> complex workflow it would be certainly large.
> > > >> I think that this is more related to how we are handling JPA in the
> > > code,
> > > >> in particular the mapping from model to entity (basically JPA is
> blind
> > > and
> > > >> has to update all nodes for every write because it believes the
> node has
> > > >> been updated, although it is not) than an issue in the table
> definition.
> > > >> In fact, when using JPA, separating the server model from the JPA
> > > entity is
> > > >> not a good idea, especially if the entity contains collections. I
> will
> > > try
> > > >> to change that without breaking anything.
> > > >>
> > > >> On Wed, Nov 22, 2023 at 12:10 PM Enrique Gonzalez Martinez <
> > > >> egonzalez@apache.org> wrote:
> > > >>
> > > >>> After the events split you now will need to create a node instance
> > > >>> model instance of making independent from the process instance.
> > > >>> That should do the trick.
> > > >>>
> > > >>> Regarding deleting/inserting it was fixed at some point.
> > > >>>
> > > >>> El mar, 21 nov 2023 a las 20:22, Francisco Javier Tirado Sarti
> > > >>> (<ft...@redhat.com>) escribió:
> > > >>> >
> > > >>> > Hi Martin,
> > > >>> > I have a task to review performance of
> > > >>> >
> > > >>> > ProcessInstanceNodeDataEventMerger
> > > >>> > My idea is to reduce the number of delete inserts when processing
> > > >>> events
> > > >>> > and try to do it incremental.
> > > >>> > That should improve performance.
> > > >>> > PS:
> > > >>> > I was planning to send an e-mail tomorrow announcing that in
> case you
> > > >>> were
> > > >>> > already working on a fix for that. I assume you are not and I
> would
> > > be
> > > >>> > sending a PR soon.
> > > >>> >
> > > >>> > On Tue, Nov 21, 2023 at 6:09 PM Martin Weiler
> > > <mweiler@ibm.com.invalid
> > > >>> >
> > > >>> > wrote:
> > > >>> >
> > > >>> > > I looked into the new examples using data-index persistence
> addon -
> > > >>> Neus'
> > > >>> > > PR#1813 [1] for serverless and Pere's branch [2] for workflow
> > > (great
> > > >>> job
> > > >>> > > both!) - and they work without issues using single requests.
> > > >>> However, under
> > > >>> > > some load (I used 'ab' for testing with a light concurrency of
> 10
> > > >>> parallel
> > > >>> > > requests) I ran into the following problems:
> > > >>> > >
> > > >>> > > (1) Large number of insert/delete calls (eg. for tables such as
> > > >>> nodes,
> > > >>> > > definitions, etc)
> > > >>> > >
> > > >>> > > (2) Hibernate OptimisticLockExceptions / StaleStateExceptions
> > > >>> > >
> > > >>> > > (3) DB deadlocks
> > > >>> > >
> > > >>> > > (4) Error responses, slow response times
> > > >>> > >
> > > >>> > > The reason I am reaching out with this topic here is to find
> out if
> > > >>> we are
> > > >>> > > aware of this issue, and if someone is already looking into or
> > > being
> > > >>> > > assigned to it?
> > > >>> > >
> > > >>> > > Thanks,
> > > >>> > > Martin
> > > >>> > >
> > > >>> > > [1]
> > > >>> https://github.com/apache/incubator-kie-kogito-examples/pull/1813
> > > >>> > > [2]
> > > >>> > >
> > > >>>
> > >
> https://github.com/pefernan/kogito-examples/tree/example_data-index_persistence
> > > >>> > >
> > > >>> > >
> > > ---------------------------------------------------------------------
> > > >>> > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > >>> > > For additional commands, e-mail: dev-help@kie.apache.org
> > > >>> > >
> > > >>> > >
> > > >>>
> > > >>>
> ---------------------------------------------------------------------
> > > >>> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > >>> For additional commands, e-mail: dev-help@kie.apache.org
> > > >>>
> > > >>>
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > For additional commands, e-mail: dev-help@kie.apache.org
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> For additional commands, e-mail: dev-help@kie.apache.org
>
>